2022-03-02 21:49:51,618 INFO MainThread:266120 [internal.py:wandb_internal():89] W&B internal server running at pid: 266120, started at: 2022-03-02 21:49:51.618058 2022-03-02 21:49:51,620 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: check_version 2022-03-02 21:49:51,620 INFO WriterThread:266120 [datastore.py:open_for_write():77] open: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/run-29cfci46.wandb 2022-03-02 21:49:51,622 DEBUG SenderThread:266120 [sender.py:send():235] send: header 2022-03-02 21:49:51,622 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: check_version 2022-03-02 21:49:51,690 DEBUG SenderThread:266120 [sender.py:send():235] send: run 2022-03-02 21:49:51,817 INFO SenderThread:266120 [dir_watcher.py:__init__():169] watching files in: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files 2022-03-02 21:49:51,817 INFO SenderThread:266120 [sender.py:_start_run_threads():809] run started: 29cfci46 with start time 1646257790 2022-03-02 21:49:51,817 DEBUG SenderThread:266120 [sender.py:send():235] send: summary 2022-03-02 21:49:51,817 INFO SenderThread:266120 [sender.py:_save_file():944] saving file wandb-summary.json with policy end 2022-03-02 21:49:51,818 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: run_start 2022-03-02 21:49:51,823 DEBUG HandlerThread:266120 [meta.py:__init__():36] meta init 2022-03-02 21:49:51,823 DEBUG HandlerThread:266120 [meta.py:__init__():50] meta init done 2022-03-02 21:49:51,823 DEBUG HandlerThread:266120 [meta.py:probe():210] probe 2022-03-02 21:49:51,829 DEBUG HandlerThread:266120 [meta.py:_setup_git():200] setup git 2022-03-02 21:49:51,844 DEBUG HandlerThread:266120 [meta.py:_setup_git():207] setup git done 2022-03-02 21:49:51,845 DEBUG HandlerThread:266120 [meta.py:_save_pip():54] save pip 2022-03-02 21:49:51,845 DEBUG HandlerThread:266120 [meta.py:_save_pip():68] save pip done 2022-03-02 21:49:51,845 DEBUG HandlerThread:266120 [meta.py:probe():248] probe done 2022-03-02 21:49:51,922 DEBUG SenderThread:266120 [sender.py:send():235] send: files 2022-03-02 21:49:51,922 INFO SenderThread:266120 [sender.py:_save_file():944] saving file wandb-metadata.json with policy now 2022-03-02 21:49:51,927 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: stop_status 2022-03-02 21:49:51,928 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: stop_status 2022-03-02 21:49:51,972 DEBUG SenderThread:266120 [sender.py:send():235] send: config 2022-03-02 21:49:51,972 DEBUG SenderThread:266120 [sender.py:send():235] send: metric 2022-03-02 21:49:51,972 DEBUG SenderThread:266120 [sender.py:send():235] send: metric 2022-03-02 21:49:51,972 WARNING SenderThread:266120 [sender.py:send_metric():902] Seen metric with glob (shouldnt happen) 2022-03-02 21:49:52,221 INFO Thread-11 :266120 [upload_job.py:push():137] Uploaded file /tmp/tmpyvsjn7qowandb/1x6pm4zs-wandb-metadata.json 2022-03-02 21:49:52,819 INFO Thread-8 :266120 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/wandb-metadata.json 2022-03-02 21:49:52,819 INFO Thread-8 :266120 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/requirements.txt 2022-03-02 21:49:52,819 INFO Thread-8 :266120 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/output.log 2022-03-02 21:49:52,819 INFO Thread-8 :266120 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/wandb-summary.json 2022-03-02 21:49:54,818 INFO Thread-8 :266120 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/output.log 2022-03-02 21:49:58,819 INFO Thread-8 :266120 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/output.log 2022-03-02 21:50:00,820 INFO Thread-8 :266120 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/output.log 2022-03-02 21:50:00,822 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 21:50:00,822 DEBUG SenderThread:266120 [sender.py:send():235] send: telemetry 2022-03-02 21:50:00,823 DEBUG SenderThread:266120 [sender.py:send():235] send: exit 2022-03-02 21:50:00,823 INFO SenderThread:266120 [sender.py:send_exit():371] handling exit code: 1 2022-03-02 21:50:00,823 INFO SenderThread:266120 [sender.py:send_exit():373] handling runtime: 9 2022-03-02 21:50:00,823 INFO SenderThread:266120 [sender.py:_save_file():944] saving file wandb-summary.json with policy end 2022-03-02 21:50:00,823 INFO SenderThread:266120 [sender.py:send_exit():379] send defer 2022-03-02 21:50:00,824 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 21:50:00,824 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: defer 2022-03-02 21:50:00,824 INFO HandlerThread:266120 [handler.py:handle_request_defer():154] handle defer: 0 2022-03-02 21:50:00,825 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: defer 2022-03-02 21:50:00,825 INFO SenderThread:266120 [sender.py:send_request_defer():388] handle sender defer: 0 2022-03-02 21:50:00,825 INFO SenderThread:266120 [sender.py:transition_state():392] send defer: 1 2022-03-02 21:50:00,825 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: defer 2022-03-02 21:50:00,825 INFO HandlerThread:266120 [handler.py:handle_request_defer():154] handle defer: 1 2022-03-02 21:50:00,863 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: defer 2022-03-02 21:50:00,863 INFO SenderThread:266120 [sender.py:send_request_defer():388] handle sender defer: 1 2022-03-02 21:50:00,863 INFO SenderThread:266120 [sender.py:transition_state():392] send defer: 2 2022-03-02 21:50:00,863 DEBUG SenderThread:266120 [sender.py:send():235] send: stats 2022-03-02 21:50:00,864 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: defer 2022-03-02 21:50:00,864 INFO HandlerThread:266120 [handler.py:handle_request_defer():154] handle defer: 2 2022-03-02 21:50:00,864 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: defer 2022-03-02 21:50:00,864 INFO SenderThread:266120 [sender.py:send_request_defer():388] handle sender defer: 2 2022-03-02 21:50:00,865 INFO SenderThread:266120 [sender.py:transition_state():392] send defer: 3 2022-03-02 21:50:00,865 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: defer 2022-03-02 21:50:00,865 INFO HandlerThread:266120 [handler.py:handle_request_defer():154] handle defer: 3 2022-03-02 21:50:00,865 DEBUG SenderThread:266120 [sender.py:send():235] send: summary 2022-03-02 21:50:00,865 INFO SenderThread:266120 [sender.py:_save_file():944] saving file wandb-summary.json with policy end 2022-03-02 21:50:00,866 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: defer 2022-03-02 21:50:00,866 INFO SenderThread:266120 [sender.py:send_request_defer():388] handle sender defer: 3 2022-03-02 21:50:00,866 INFO SenderThread:266120 [sender.py:transition_state():392] send defer: 4 2022-03-02 21:50:00,866 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: defer 2022-03-02 21:50:00,866 INFO HandlerThread:266120 [handler.py:handle_request_defer():154] handle defer: 4 2022-03-02 21:50:00,866 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: defer 2022-03-02 21:50:00,866 INFO SenderThread:266120 [sender.py:send_request_defer():388] handle sender defer: 4 2022-03-02 21:50:00,925 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 21:50:00,982 INFO SenderThread:266120 [sender.py:transition_state():392] send defer: 5 2022-03-02 21:50:00,983 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 21:50:00,983 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: defer 2022-03-02 21:50:00,983 INFO HandlerThread:266120 [handler.py:handle_request_defer():154] handle defer: 5 2022-03-02 21:50:00,984 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: defer 2022-03-02 21:50:00,984 INFO SenderThread:266120 [sender.py:send_request_defer():388] handle sender defer: 5 2022-03-02 21:50:00,984 INFO SenderThread:266120 [dir_watcher.py:finish():283] shutting down directory watcher 2022-03-02 21:50:01,084 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 21:50:01,821 INFO Thread-8 :266120 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/wandb-summary.json 2022-03-02 21:50:01,821 INFO SenderThread:266120 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/output.log 2022-03-02 21:50:01,821 INFO SenderThread:266120 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/config.yaml 2022-03-02 21:50:01,821 INFO SenderThread:266120 [dir_watcher.py:finish():313] scan: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files 2022-03-02 21:50:01,822 INFO SenderThread:266120 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/wandb-metadata.json wandb-metadata.json 2022-03-02 21:50:01,822 INFO SenderThread:266120 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/output.log output.log 2022-03-02 21:50:01,822 INFO SenderThread:266120 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/wandb-summary.json wandb-summary.json 2022-03-02 21:50:01,825 INFO SenderThread:266120 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/requirements.txt requirements.txt 2022-03-02 21:50:01,828 INFO SenderThread:266120 [dir_watcher.py:finish():327] scan save: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/config.yaml config.yaml 2022-03-02 21:50:01,828 INFO SenderThread:266120 [sender.py:transition_state():392] send defer: 6 2022-03-02 21:50:01,828 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 21:50:01,835 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: defer 2022-03-02 21:50:01,835 INFO HandlerThread:266120 [handler.py:handle_request_defer():154] handle defer: 6 2022-03-02 21:50:01,836 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: defer 2022-03-02 21:50:01,836 INFO SenderThread:266120 [sender.py:send_request_defer():388] handle sender defer: 6 2022-03-02 21:50:01,836 INFO SenderThread:266120 [file_pusher.py:finish():177] shutting down file pusher 2022-03-02 21:50:01,934 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 21:50:01,934 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 21:50:02,036 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 21:50:02,036 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 21:50:02,085 INFO Thread-13 :266120 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/wandb-summary.json 2022-03-02 21:50:02,091 INFO Thread-15 :266120 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/config.yaml 2022-03-02 21:50:02,118 INFO Thread-12 :266120 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/output.log 2022-03-02 21:50:02,127 INFO Thread-14 :266120 [upload_job.py:push():137] Uploaded file /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/files/requirements.txt 2022-03-02 21:50:02,138 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 21:50:02,138 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 21:50:02,240 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 21:50:02,240 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 21:50:02,327 INFO Thread-7 :266120 [sender.py:transition_state():392] send defer: 7 2022-03-02 21:50:02,328 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: defer 2022-03-02 21:50:02,328 INFO HandlerThread:266120 [handler.py:handle_request_defer():154] handle defer: 7 2022-03-02 21:50:02,328 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: defer 2022-03-02 21:50:02,328 INFO SenderThread:266120 [sender.py:send_request_defer():388] handle sender defer: 7 2022-03-02 21:50:02,342 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 21:50:02,381 INFO SenderThread:266120 [sender.py:transition_state():392] send defer: 8 2022-03-02 21:50:02,381 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 21:50:02,382 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: defer 2022-03-02 21:50:02,382 INFO HandlerThread:266120 [handler.py:handle_request_defer():154] handle defer: 8 2022-03-02 21:50:02,382 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: defer 2022-03-02 21:50:02,382 INFO SenderThread:266120 [sender.py:send_request_defer():388] handle sender defer: 8 2022-03-02 21:50:02,382 INFO SenderThread:266120 [sender.py:transition_state():392] send defer: 9 2022-03-02 21:50:02,383 DEBUG SenderThread:266120 [sender.py:send():235] send: final 2022-03-02 21:50:02,383 DEBUG SenderThread:266120 [sender.py:send():235] send: footer 2022-03-02 21:50:02,383 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: defer 2022-03-02 21:50:02,383 INFO HandlerThread:266120 [handler.py:handle_request_defer():154] handle defer: 9 2022-03-02 21:50:02,383 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: defer 2022-03-02 21:50:02,383 INFO SenderThread:266120 [sender.py:send_request_defer():388] handle sender defer: 9 2022-03-02 21:50:02,483 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: poll_exit 2022-03-02 21:50:02,483 DEBUG SenderThread:266120 [sender.py:send_request():249] send_request: poll_exit 2022-03-02 21:50:02,483 INFO SenderThread:266120 [file_pusher.py:join():182] waiting for file pusher 2022-03-02 21:50:02,547 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: get_summary 2022-03-02 21:50:02,548 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: sampled_history 2022-03-02 21:50:02,548 DEBUG HandlerThread:266120 [handler.py:handle_request():131] handle_request: shutdown 2022-03-02 21:50:02,548 INFO HandlerThread:266120 [handler.py:finish():739] shutting down handler 2022-03-02 21:50:03,383 INFO WriterThread:266120 [datastore.py:close():281] close: /home/sanchit_huggingface_co/wav2vec2-2-rnd-grid-search/wandb/run-20220302_214950-29cfci46/run-29cfci46.wandb 2022-03-02 21:50:03,546 INFO SenderThread:266120 [sender.py:finish():1075] shutting down sender 2022-03-02 21:50:03,546 INFO SenderThread:266120 [file_pusher.py:finish():177] shutting down file pusher 2022-03-02 21:50:03,546 INFO SenderThread:266120 [file_pusher.py:join():182] waiting for file pusher 2022-03-02 21:50:03,548 INFO MainThread:266120 [internal.py:handle_exit():79] Internal process exited