2024-11-13 19:02:46,431 INFO StreamThr :1939 [internal.py:wandb_internal():86] W&B internal server running at pid: 1939, started at: 2024-11-13 19:02:46.431351 2024-11-13 19:02:46,433 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status 2024-11-13 19:02:46,434 INFO WriterThread:1939 [datastore.py:open_for_write():87] open: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/run-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1.wandb 2024-11-13 19:02:46,435 DEBUG SenderThread:1939 [sender.py:send():382] send: header 2024-11-13 19:02:46,442 DEBUG SenderThread:1939 [sender.py:send():382] send: run 2024-11-13 19:02:46,772 INFO SenderThread:1939 [dir_watcher.py:__init__():211] watching files in: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files 2024-11-13 19:02:46,772 INFO SenderThread:1939 [sender.py:_start_run_threads():1136] run started: kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1 with start time 1731524566.431221 2024-11-13 19:02:46,782 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: check_version 2024-11-13 19:02:46,782 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: check_version 2024-11-13 19:02:46,859 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: run_start 2024-11-13 19:02:46,947 DEBUG HandlerThread:1939 [system_info.py:__init__():27] System info init 2024-11-13 19:02:46,947 DEBUG HandlerThread:1939 [system_info.py:__init__():42] System info init done 2024-11-13 19:02:46,947 INFO HandlerThread:1939 [system_monitor.py:start():194] Starting system monitor 2024-11-13 19:02:46,947 INFO SystemMonitor:1939 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-11-13 19:02:46,948 INFO HandlerThread:1939 [system_monitor.py:probe():214] Collecting system info 2024-11-13 19:02:46,948 INFO SystemMonitor:1939 [interfaces.py:start():190] Started cpu monitoring 2024-11-13 19:02:46,949 INFO SystemMonitor:1939 [interfaces.py:start():190] Started disk monitoring 2024-11-13 19:02:46,949 INFO SystemMonitor:1939 [interfaces.py:start():190] Started gpu monitoring 2024-11-13 19:02:46,951 INFO SystemMonitor:1939 [interfaces.py:start():190] Started memory monitoring 2024-11-13 19:02:46,952 INFO SystemMonitor:1939 [interfaces.py:start():190] Started network monitoring 2024-11-13 19:02:47,010 DEBUG HandlerThread:1939 [system_info.py:probe():151] Probing system 2024-11-13 19:02:47,013 DEBUG HandlerThread:1939 [system_info.py:_probe_git():136] Probing git 2024-11-13 19:02:47,021 DEBUG HandlerThread:1939 [system_info.py:_probe_git():144] Probing git done 2024-11-13 19:02:47,021 DEBUG HandlerThread:1939 [system_info.py:probe():199] Probing system done 2024-11-13 19:02:47,022 DEBUG HandlerThread:1939 [system_monitor.py:probe():223] {'os': 'Linux-5.10.226-214.879.amzn2.x86_64-x86_64-with-glibc2.31', 'python': '3.10.12', 'heartbeatAt': '2024-11-13T19:02:47.010671', 'startedAt': '2024-11-13T19:02:46.426230', 'docker': None, 'cuda': None, 'args': ('--batch_size', '64', '--cache_dir', '/opt/ml/data/input/.cache', '--dataset', 'code-contests', '--gradient_accumulation_steps', '8', '--logging_steps', '5', '--lr', '1e-6', '--max_seq_length', '2048', '--model_name', 'Qwen/Qwen2.5-Coder-7B-Instruct', '--num_train_epochs', '2', '--output_dir', '/opt/ml/model/', '--push_to_hub', 'True', '--run_name', 'sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048', '--save_steps', '100', '--schedule', 'cosine', '--wandb_project', 'sft-codecontests-1112', '--weight_decay', '0.0'), 'state': 'running', 'program': '/opt/ml/code/qlearning_reasoning/training/sft.py', 'codePathLocal': 'qlearning_reasoning/training/sft.py', 'codePath': 'qlearning_reasoning/training/sft.py', 'git': {'remote': 'git@github.com:TRI-ML/reasoning-value-verifiers.git', 'commit': 'cbfd05147f15c55be83f4996eebdb305c17f119d'}, 'email': None, 'root': '/opt/ml/code', 'host': 'algo-1', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 96, 'cpu_count_logical': 192, 'cpu_freq': {'current': 2829.910838541667, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2850.788, 'min': 0.0, 'max': 0.0}, {'current': 3406.159, 'min': 0.0, 'max': 0.0}, {'current': 2605.944, 'min': 0.0, 'max': 0.0}, {'current': 2608.162, 'min': 0.0, 'max': 0.0}, {'current': 2818.169, 'min': 0.0, 'max': 0.0}, {'current': 2591.551, 'min': 0.0, 'max': 0.0}, {'current': 2621.486, 'min': 0.0, 'max': 0.0}, {'current': 2607.504, 'min': 0.0, 'max': 0.0}, {'current': 2999.734, 'min': 0.0, 'max': 0.0}, {'current': 2991.449, 'min': 0.0, 'max': 0.0}, {'current': 3123.177, 'min': 0.0, 'max': 0.0}, {'current': 3053.641, 'min': 0.0, 'max': 0.0}, {'current': 3003.014, 'min': 0.0, 'max': 0.0}, {'current': 3000.178, 'min': 0.0, 'max': 0.0}, {'current': 3432.256, 'min': 0.0, 'max': 0.0}, {'current': 2999.802, 'min': 0.0, 'max': 0.0}, {'current': 3005.725, 'min': 0.0, 'max': 0.0}, {'current': 3565.885, 'min': 0.0, 'max': 0.0}, {'current': 2992.339, 'min': 0.0, 'max': 0.0}, {'current': 3104.468, 'min': 0.0, 'max': 0.0}, {'current': 3182.411, 'min': 0.0, 'max': 0.0}, {'current': 3013.217, 'min': 0.0, 'max': 0.0}, {'current': 3131.904, 'min': 0.0, 'max': 0.0}, {'current': 3007.314, 'min': 0.0, 'max': 0.0}, {'current': 2625.175, 'min': 0.0, 'max': 0.0}, {'current': 2653.407, 'min': 0.0, 'max': 0.0}, {'current': 2610.871, 'min': 0.0, 'max': 0.0}, {'current': 2793.855, 'min': 0.0, 'max': 0.0}, {'current': 2619.232, 'min': 0.0, 'max': 0.0}, {'current': 2627.886, 'min': 0.0, 'max': 0.0}, {'current': 2613.748, 'min': 0.0, 'max': 0.0}, {'current': 2318.039, 'min': 0.0, 'max': 0.0}, {'current': 3546.661, 'min': 0.0, 'max': 0.0}, {'current': 3022.56, 'min': 0.0, 'max': 0.0}, {'current': 3035.28, 'min': 0.0, 'max': 0.0}, {'current': 3541.443, 'min': 0.0, 'max': 0.0}, {'current': 3163.55, 'min': 0.0, 'max': 0.0}, {'current': 3023.79, 'min': 0.0, 'max': 0.0}, {'current': 3033.204, 'min': 0.0, 'max': 0.0}, {'current': 3031.695, 'min': 0.0, 'max': 0.0}, {'current': 3034.587, 'min': 0.0, 'max': 0.0}, {'current': 3031.71, 'min': 0.0, 'max': 0.0}, {'current': 3029.116, 'min': 0.0, 'max': 0.0}, {'current': 3293.701, 'min': 0.0, 'max': 0.0}, {'current': 3033.78, 'min': 0.0, 'max': 0.0}, {'current': 3032.034, 'min': 0.0, 'max': 0.0}, {'current': 3057.365, 'min': 0.0, 'max': 0.0}, {'current': 3028.183, 'min': 0.0, 'max': 0.0}, {'current': 3596.439, 'min': 0.0, 'max': 0.0}, {'current': 3327.853, 'min': 0.0, 'max': 0.0}, {'current': 3226.463, 'min': 0.0, 'max': 0.0}, {'current': 3259.361, 'min': 0.0, 'max': 0.0}, {'current': 3284.076, 'min': 0.0, 'max': 0.0}, {'current': 3257.698, 'min': 0.0, 'max': 0.0}, {'current': 3280.289, 'min': 0.0, 'max': 0.0}, {'current': 3258.834, 'min': 0.0, 'max': 0.0}, {'current': 2517.346, 'min': 0.0, 'max': 0.0}, {'current': 2553.091, 'min': 0.0, 'max': 0.0}, {'current': 2552.693, 'min': 0.0, 'max': 0.0}, {'current': 2689.091, 'min': 0.0, 'max': 0.0}, {'current': 2503.301, 'min': 0.0, 'max': 0.0}, {'current': 2524.4, 'min': 0.0, 'max': 0.0}, {'current': 2521.768, 'min': 0.0, 'max': 0.0}, {'current': 2530.364, 'min': 0.0, 'max': 0.0}, {'current': 3258.225, 'min': 0.0, 'max': 0.0}, {'current': 3597.406, 'min': 0.0, 'max': 0.0}, {'current': 3279.658, 'min': 0.0, 'max': 0.0}, {'current': 3292.257, 'min': 0.0, 'max': 0.0}, {'current': 3265.351, 'min': 0.0, 'max': 0.0}, {'current': 3286.715, 'min': 0.0, 'max': 0.0}, {'current': 3268.777, 'min': 0.0, 'max': 0.0}, {'current': 3242.068, 'min': 0.0, 'max': 0.0}, {'current': 3288.908, 'min': 0.0, 'max': 0.0}, {'current': 3278.645, 'min': 0.0, 'max': 0.0}, {'current': 3215.578, 'min': 0.0, 'max': 0.0}, {'current': 3327.101, 'min': 0.0, 'max': 0.0}, {'current': 3357.075, 'min': 0.0, 'max': 0.0}, {'current': 3144.838, 'min': 0.0, 'max': 0.0}, {'current': 3005.517, 'min': 0.0, 'max': 0.0}, {'current': 3597.381, 'min': 0.0, 'max': 0.0}, {'current': 3596.175, 'min': 0.0, 'max': 0.0}, {'current': 3317.258, 'min': 0.0, 'max': 0.0}, {'current': 3310.249, 'min': 0.0, 'max': 0.0}, {'current': 3140.687, 'min': 0.0, 'max': 0.0}, {'current': 3398.954, 'min': 0.0, 'max': 0.0}, {'current': 3318.433, 'min': 0.0, 'max': 0.0}, {'current': 3342.35, 'min': 0.0, 'max': 0.0}, {'current': 3328.575, 'min': 0.0, 'max': 0.0}, {'current': 3499.892, 'min': 0.0, 'max': 0.0}, {'current': 3344.864, 'min': 0.0, 'max': 0.0}, {'current': 3355.802, 'min': 0.0, 'max': 0.0}, {'current': 3305.032, 'min': 0.0, 'max': 0.0}, {'current': 3318.07, 'min': 0.0, 'max': 0.0}, {'current': 3336.546, 'min': 0.0, 'max': 0.0}, {'current': 3335.458, 'min': 0.0, 'max': 0.0}, {'current': 3596.561, 'min': 0.0, 'max': 0.0}, {'current': 2884.473, 'min': 0.0, 'max': 0.0}, {'current': 2934.098, 'min': 0.0, 'max': 0.0}, {'current': 2621.598, 'min': 0.0, 'max': 0.0}, {'current': 2645.184, 'min': 0.0, 'max': 0.0}, {'current': 2480.726, 'min': 0.0, 'max': 0.0}, {'current': 2515.912, 'min': 0.0, 'max': 0.0}, {'current': 2702.57, 'min': 0.0, 'max': 0.0}, {'current': 2671.473, 'min': 0.0, 'max': 0.0}, {'current': 3092.321, 'min': 0.0, 'max': 0.0}, {'current': 3092.204, 'min': 0.0, 'max': 0.0}, {'current': 3217.287, 'min': 0.0, 'max': 0.0}, {'current': 3161.646, 'min': 0.0, 'max': 0.0}, {'current': 3089.273, 'min': 0.0, 'max': 0.0}, {'current': 3088.447, 'min': 0.0, 'max': 0.0}, {'current': 3559.214, 'min': 0.0, 'max': 0.0}, {'current': 3218.611, 'min': 0.0, 'max': 0.0}, {'current': 3021.622, 'min': 0.0, 'max': 0.0}, {'current': 3332.084, 'min': 0.0, 'max': 0.0}, {'current': 3019.854, 'min': 0.0, 'max': 0.0}, {'current': 3213.831, 'min': 0.0, 'max': 0.0}, {'current': 3092.774, 'min': 0.0, 'max': 0.0}, {'current': 3015.868, 'min': 0.0, 'max': 0.0}, {'current': 3082.752, 'min': 0.0, 'max': 0.0}, {'current': 3029.559, 'min': 0.0, 'max': 0.0}, {'current': 2873.142, 'min': 0.0, 'max': 0.0}, {'current': 2215.288, 'min': 0.0, 'max': 0.0}, {'current': 3193.443, 'min': 0.0, 'max': 0.0}, {'current': 2928.717, 'min': 0.0, 'max': 0.0}, {'current': 2990.366, 'min': 0.0, 'max': 0.0}, {'current': 2923.847, 'min': 0.0, 'max': 0.0}, {'current': 2869.364, 'min': 0.0, 'max': 0.0}, {'current': 2859.189, 'min': 0.0, 'max': 0.0}, {'current': 3049.001, 'min': 0.0, 'max': 0.0}, {'current': 2805.206, 'min': 0.0, 'max': 0.0}, {'current': 3094.081, 'min': 0.0, 'max': 0.0}, {'current': 3187.222, 'min': 0.0, 'max': 0.0}, {'current': 2853.945, 'min': 0.0, 'max': 0.0}, {'current': 2944.73, 'min': 0.0, 'max': 0.0}, {'current': 2809.161, 'min': 0.0, 'max': 0.0}, {'current': 2806.787, 'min': 0.0, 'max': 0.0}, {'current': 3024.225, 'min': 0.0, 'max': 0.0}, {'current': 3032.173, 'min': 0.0, 'max': 0.0}, {'current': 3033.195, 'min': 0.0, 'max': 0.0}, {'current': 3561.287, 'min': 0.0, 'max': 0.0}, {'current': 2987.872, 'min': 0.0, 'max': 0.0}, {'current': 3051.252, 'min': 0.0, 'max': 0.0}, {'current': 3379.017, 'min': 0.0, 'max': 0.0}, {'current': 3044.319, 'min': 0.0, 'max': 0.0}, {'current': 3595.62, 'min': 0.0, 'max': 0.0}, {'current': 3261.737, 'min': 0.0, 'max': 0.0}, {'current': 3257.013, 'min': 0.0, 'max': 0.0}, {'current': 3288.317, 'min': 0.0, 'max': 0.0}, {'current': 3268.675, 'min': 0.0, 'max': 0.0}, {'current': 3269.579, 'min': 0.0, 'max': 0.0}, {'current': 3266.213, 'min': 0.0, 'max': 0.0}, {'current': 3265.148, 'min': 0.0, 'max': 0.0}, {'current': 2554.152, 'min': 0.0, 'max': 0.0}, {'current': 2556.63, 'min': 0.0, 'max': 0.0}, {'current': 2540.571, 'min': 0.0, 'max': 0.0}, {'current': 2512.005, 'min': 0.0, 'max': 0.0}, {'current': 2560.992, 'min': 0.0, 'max': 0.0}, {'current': 2526.641, 'min': 0.0, 'max': 0.0}, {'current': 2528.267, 'min': 0.0, 'max': 0.0}, {'current': 2583.061, 'min': 0.0, 'max': 0.0}, {'current': 3257.652, 'min': 0.0, 'max': 0.0}, {'current': 3598.217, 'min': 0.0, 'max': 0.0}, {'current': 3282.578, 'min': 0.0, 'max': 0.0}, {'current': 3261.664, 'min': 0.0, 'max': 0.0}, {'current': 3268.246, 'min': 0.0, 'max': 0.0}, {'current': 3331.937, 'min': 0.0, 'max': 0.0}, {'current': 3265.603, 'min': 0.0, 'max': 0.0}, {'current': 3258.501, 'min': 0.0, 'max': 0.0}, {'current': 3337.045, 'min': 0.0, 'max': 0.0}, {'current': 3259.949, 'min': 0.0, 'max': 0.0}, {'current': 3226.058, 'min': 0.0, 'max': 0.0}, {'current': 3250.308, 'min': 0.0, 'max': 0.0}, {'current': 3208.026, 'min': 0.0, 'max': 0.0}, {'current': 2974.043, 'min': 0.0, 'max': 0.0}, {'current': 3216.706, 'min': 0.0, 'max': 0.0}, {'current': 3598.55, 'min': 0.0, 'max': 0.0}, {'current': 3598.261, 'min': 0.0, 'max': 0.0}, {'current': 3222.637, 'min': 0.0, 'max': 0.0}, {'current': 3227.662, 'min': 0.0, 'max': 0.0}, {'current': 3143.781, 'min': 0.0, 'max': 0.0}, {'current': 3215.567, 'min': 0.0, 'max': 0.0}, {'current': 3250.612, 'min': 0.0, 'max': 0.0}, {'current': 3195.784, 'min': 0.0, 'max': 0.0}, {'current': 3219.289, 'min': 0.0, 'max': 0.0}, {'current': 3289.163, 'min': 0.0, 'max': 0.0}, {'current': 3247.875, 'min': 0.0, 'max': 0.0}, {'current': 3249.112, 'min': 0.0, 'max': 0.0}, {'current': 3209.3, 'min': 0.0, 'max': 0.0}, {'current': 3187.212, 'min': 0.0, 'max': 0.0}, {'current': 3288.003, 'min': 0.0, 'max': 0.0}, {'current': 3169.417, 'min': 0.0, 'max': 0.0}, {'current': 3596.071, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 119.94140625, 'used': 56.4015998840332}}, 'gpu': 'NVIDIA H100 80GB HBM3', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}], 'memory': {'total': 1999.9661331176758}} 2024-11-13 19:02:47,022 INFO HandlerThread:1939 [system_monitor.py:probe():224] Finished collecting system info 2024-11-13 19:02:47,022 INFO HandlerThread:1939 [system_monitor.py:probe():227] Publishing system info 2024-11-13 19:02:47,022 DEBUG HandlerThread:1939 [system_info.py:_save_conda():208] Saving list of conda packages installed into the current environment 2024-11-13 19:02:47,775 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/conda-environment.yaml 2024-11-13 19:03:01,345 DEBUG HandlerThread:1939 [system_info.py:_save_conda():220] Saving conda packages done 2024-11-13 19:03:01,345 DEBUG HandlerThread:1939 [system_info.py:_save_code():45] Saving code 2024-11-13 19:03:01,349 DEBUG HandlerThread:1939 [system_info.py:_save_code():66] Saving code done 2024-11-13 19:03:01,349 DEBUG HandlerThread:1939 [system_info.py:_save_patches():83] Saving git patches 2024-11-13 19:03:01,776 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/conda-environment.yaml 2024-11-13 19:03:01,776 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/code/qlearning_reasoning/training/sft.py 2024-11-13 19:03:01,776 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/code/qlearning_reasoning 2024-11-13 19:03:01,776 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/code/qlearning_reasoning/training 2024-11-13 19:03:01,776 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/code 2024-11-13 19:03:01,989 DEBUG HandlerThread:1939 [system_info.py:_save_patches():125] Saving git patches done 2024-11-13 19:03:01,991 INFO HandlerThread:1939 [system_monitor.py:probe():229] Finished publishing system info 2024-11-13 19:03:01,993 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:01,993 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: keepalive 2024-11-13 19:03:01,994 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:01,994 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: keepalive 2024-11-13 19:03:01,994 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:01,994 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: keepalive 2024-11-13 19:03:01,994 DEBUG SenderThread:1939 [sender.py:send():382] send: files 2024-11-13 19:03:01,995 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now 2024-11-13 19:03:01,995 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file code/qlearning_reasoning/training/sft.py with policy now 2024-11-13 19:03:01,995 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file diff.patch with policy now 2024-11-13 19:03:01,999 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: python_packages 2024-11-13 19:03:01,999 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: python_packages 2024-11-13 19:03:02,001 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:03:02,003 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:03:02,003 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:03:02,111 DEBUG SenderThread:1939 [sender.py:send():382] send: telemetry 2024-11-13 19:03:02,111 DEBUG SenderThread:1939 [sender.py:send():382] send: config 2024-11-13 19:03:02,112 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 19:03:02,112 DEBUG SenderThread:1939 [sender.py:send():382] send: telemetry 2024-11-13 19:03:02,112 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 19:03:02,113 WARNING SenderThread:1939 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) 2024-11-13 19:03:02,113 DEBUG SenderThread:1939 [sender.py:send():382] send: telemetry 2024-11-13 19:03:02,113 DEBUG SenderThread:1939 [sender.py:send():382] send: telemetry 2024-11-13 19:03:02,113 DEBUG SenderThread:1939 [sender.py:send():382] send: config 2024-11-13 19:03:02,551 INFO wandb-upload_1:1939 [upload_job.py:push():131] Uploaded file /tmp/tmpm_h7h9v0wandb/pjhigh2e-code/qlearning_reasoning/training/sft.py 2024-11-13 19:03:02,554 INFO wandb-upload_0:1939 [upload_job.py:push():131] Uploaded file /tmp/tmpm_h7h9v0wandb/f98paeu8-wandb-metadata.json 2024-11-13 19:03:02,556 INFO wandb-upload_2:1939 [upload_job.py:push():131] Uploaded file /tmp/tmpm_h7h9v0wandb/cc8zuviy-diff.patch 2024-11-13 19:03:02,776 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/diff.patch 2024-11-13 19:03:02,776 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:03:02,777 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/requirements.txt 2024-11-13 19:03:02,777 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/wandb-metadata.json 2024-11-13 19:03:04,777 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:03:07,114 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:12,115 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:16,999 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:03:17,000 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:03:17,002 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:03:18,090 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:18,782 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/config.yaml 2024-11-13 19:03:23,250 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:26,095 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 19:03:26,097 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 19:03:26,097 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 19:03:26,097 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 19:03:26,097 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 19:03:26,097 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 19:03:26,098 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:03:26,099 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:03:26,785 INFO Thread-12 :1939 [dir_watcher.py:_on_file_created():271] file/dir created: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/wandb-summary.json 2024-11-13 19:03:28,785 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:03:29,100 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:31,999 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:03:32,000 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:03:32,002 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:03:34,147 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:39,147 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:44,148 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:46,955 DEBUG SystemMonitor:1939 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-11-13 19:03:46,959 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:03:47,000 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:03:47,000 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:03:47,043 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:03:49,636 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:50,813 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:03:50,814 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/config.yaml 2024-11-13 19:03:54,814 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:03:59,814 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:02,000 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:04:02,000 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:04:02,002 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:04:05,061 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:10,062 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:14,822 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:04:15,419 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:16,962 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:04:17,000 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:04:17,000 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:04:17,002 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:04:21,087 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:26,088 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:31,089 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:32,000 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:04:32,000 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:04:32,043 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:04:36,587 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:36,829 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:04:41,588 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:46,588 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:46,965 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:04:47,000 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:04:47,000 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:04:47,002 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:04:52,111 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:57,111 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:04:57,383 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 19:04:57,384 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 19:04:57,385 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:04:57,385 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:04:57,837 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/wandb-summary.json 2024-11-13 19:04:58,837 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:04:59,002 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 19:04:59,004 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 19:04:59,005 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 19:04:59,005 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 19:04:59,005 DEBUG SenderThread:1939 [sender.py:send():382] send: metric 2024-11-13 19:04:59,006 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 19:04:59,006 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:04:59,006 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:04:59,838 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/wandb-summary.json 2024-11-13 19:05:00,838 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:05:02,000 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:05:02,001 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:05:02,002 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:05:03,111 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:05:08,112 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:05:13,113 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:05:16,967 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:05:17,000 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:05:17,001 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:05:17,043 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:05:18,123 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:05:22,846 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:05:23,982 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:05:25,847 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/config.yaml 2024-11-13 19:05:30,486 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:05:32,000 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:05:32,001 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:05:32,002 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:05:36,117 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:05:41,118 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:05:46,513 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:05:46,854 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:05:46,970 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:05:47,000 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:05:47,001 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:05:47,003 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:05:52,097 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:05:57,098 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:02,001 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:06:02,001 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:06:02,043 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:06:02,152 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:07,153 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:08,862 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:06:12,639 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:16,973 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:06:17,001 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:06:17,002 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:06:17,004 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:06:18,127 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:23,127 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:28,128 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:30,870 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:06:32,001 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:06:32,001 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:06:32,003 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:06:33,143 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:38,144 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:43,144 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:46,975 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:06:47,001 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:06:47,001 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:06:47,043 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:06:49,137 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:51,931 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 19:06:51,932 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 19:06:51,933 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:06:51,934 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:06:52,878 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:06:52,878 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/wandb-summary.json 2024-11-13 19:06:53,541 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 19:06:53,543 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 19:06:53,543 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:06:53,544 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:06:53,878 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/wandb-summary.json 2024-11-13 19:06:54,545 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:06:54,878 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:06:59,546 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:07:02,001 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:07:02,002 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:07:02,004 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:07:05,164 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:07:10,165 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:07:15,165 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:07:16,886 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:07:16,978 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:07:17,001 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:07:17,002 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:07:17,003 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:07:21,108 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:07:26,108 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:07:31,109 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:07:32,001 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:07:32,002 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:07:32,043 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:07:36,934 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:07:38,894 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:07:41,935 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:07:46,935 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:07:46,981 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:07:47,002 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:07:47,002 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:07:47,004 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:07:52,135 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:07:57,136 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:02,136 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:02,856 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:08:02,923 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:08:02,924 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:08:08,076 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:13,077 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:16,985 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:08:17,407 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:08:17,407 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:08:17,447 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:08:18,545 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:23,546 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:28,546 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:32,914 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:08:33,843 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:34,729 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:08:34,729 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:08:34,771 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:08:39,802 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:44,803 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:46,987 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:08:48,007 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:08:48,007 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:08:48,008 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:08:48,921 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:08:49,945 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:50,922 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:08:52,923 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:08:54,924 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:08:55,154 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:08:56,925 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:08:58,926 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:09:00,371 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:09:00,948 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:09:03,950 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:09:05,541 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:09:05,782 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:09:06,102 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:09:06,102 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:09:11,263 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:09:16,264 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:09:16,990 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:09:20,961 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:09:22,073 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:09:22,272 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:09:22,272 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:09:27,401 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:09:32,401 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:09:36,126 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:09:36,666 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:09:36,666 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:09:38,052 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:09:39,964 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:09:43,769 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:09:46,992 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:09:47,846 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:09:47,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:09:47,847 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:09:49,230 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:09:54,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:09:59,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:10:02,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:10:02,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:10:02,848 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:10:05,230 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:10:10,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:10:11,977 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:10:15,248 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:10:16,996 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:10:17,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:10:17,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:10:17,848 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:10:21,230 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:10:26,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:10:31,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:10:32,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:10:32,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:10:32,847 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:10:33,985 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:10:37,230 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:10:42,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:10:46,998 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:10:47,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:10:47,847 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:10:47,887 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:10:47,921 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:10:53,230 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:10:53,575 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 19:10:53,576 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 19:10:53,576 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:10:53,577 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:10:53,992 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/wandb-summary.json 2024-11-13 19:10:54,993 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:10:55,200 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 19:10:55,201 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 19:10:55,201 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:10:55,203 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:10:55,993 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:10:55,994 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/wandb-summary.json 2024-11-13 19:10:56,994 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:10:57,994 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:10:59,204 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:11:02,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:11:02,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:11:02,849 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:11:04,230 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:11:09,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:11:14,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:11:17,000 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:11:17,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:11:17,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:11:17,848 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:11:19,311 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:11:20,004 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:11:24,312 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:11:29,312 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:11:32,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:11:32,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:11:32,849 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:11:35,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:11:40,232 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:11:44,014 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:11:46,173 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:11:47,003 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:11:47,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:11:47,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:11:47,848 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:11:51,230 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:11:56,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:01,232 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:02,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:12:02,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:12:02,848 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:12:06,021 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:12:07,202 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:12,203 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:17,006 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:12:17,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:12:17,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:12:17,848 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:12:17,991 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:23,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:28,029 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:12:28,901 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:32,847 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:12:32,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:12:32,848 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:12:34,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:39,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:44,232 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:47,008 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:12:47,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:12:47,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:12:47,848 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:12:48,871 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 19:12:48,872 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 19:12:48,873 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:12:48,873 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:12:49,037 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/wandb-summary.json 2024-11-13 19:12:49,401 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:50,038 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:12:50,481 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 19:12:50,482 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 19:12:50,483 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:12:50,484 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:12:51,038 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:12:51,039 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/wandb-summary.json 2024-11-13 19:12:52,039 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:12:54,485 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:12:59,486 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:13:02,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:13:02,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:13:02,848 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:13:05,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:13:10,232 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:13:14,047 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:13:15,403 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:13:17,011 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:13:17,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:13:17,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:13:17,849 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:13:21,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:13:26,232 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:13:31,232 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:13:32,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:13:32,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:13:32,849 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:13:36,055 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:13:36,280 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:13:41,280 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:13:46,281 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:13:47,014 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:13:47,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:13:47,848 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:13:47,849 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:13:52,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:13:57,232 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:14:00,064 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:14:02,559 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:14:02,854 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:14:02,854 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:14:02,856 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:14:08,232 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:14:13,232 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:14:17,017 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:14:17,855 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:14:17,855 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:14:17,895 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:14:19,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:14:22,072 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:14:24,329 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:14:29,330 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:14:32,854 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:14:32,855 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:14:32,855 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:14:35,231 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:14:40,232 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:14:45,232 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:14:47,019 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:14:48,157 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:14:48,157 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:14:48,415 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:14:50,256 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:14:56,051 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:01,051 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:02,910 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:15:02,910 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:15:03,032 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:15:06,239 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:10,089 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:15:11,478 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:16,478 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:17,022 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:15:18,866 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:15:18,866 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:15:19,390 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:15:22,355 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:27,579 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:28,095 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:15:30,096 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:15:32,097 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:15:32,786 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:32,973 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:15:32,973 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:15:33,126 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:15:34,098 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:15:36,099 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:15:37,917 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:38,100 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:15:40,101 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:15:43,713 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:47,025 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:15:49,026 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:49,048 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:15:49,666 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:15:49,666 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:15:54,756 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:15:59,756 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:16:03,643 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:16:03,643 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:16:04,015 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:16:05,465 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:16:10,466 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:16:15,937 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:16:16,113 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:16:17,030 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:16:18,245 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:16:18,245 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:16:18,549 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:16:21,587 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:16:26,588 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:16:31,588 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:16:33,245 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:16:33,246 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:16:33,549 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:16:37,587 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:16:42,588 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:16:47,030 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:16:48,032 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:16:48,251 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:16:48,251 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:16:49,121 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:16:53,293 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:16:58,293 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:03,294 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:05,806 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:17:05,806 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:17:05,941 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:17:08,940 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:13,941 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:17,033 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:17:19,035 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:20,127 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:17:20,128 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:17:20,171 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:17:24,246 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:30,141 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:17:30,209 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:35,210 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:35,928 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:17:35,928 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:17:36,794 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:17:40,795 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:45,796 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:47,036 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:17:51,038 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:51,987 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:17:51,987 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:17:51,988 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:17:54,152 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:17:56,164 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:17:56,204 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:17:59,166 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:18:01,167 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:18:01,358 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:18:03,168 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:18:05,169 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:18:07,169 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:18:07,346 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:18:12,347 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:18:13,267 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:18:13,476 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:18:13,476 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:18:17,039 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:18:18,041 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:18:23,042 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:18:28,199 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:18:29,298 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:18:30,038 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:18:30,039 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:18:34,183 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:18:39,184 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:18:43,601 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:18:43,602 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:18:44,627 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:18:44,740 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:18:47,042 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:18:50,044 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:18:53,037 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:18:53,037 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: partial_history 2024-11-13 19:18:53,039 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:18:53,039 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:18:53,040 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:18:53,040 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:18:53,040 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:18:53,040 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:18:53,040 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:18:53,041 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:18:53,041 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:18:53,041 DEBUG SenderThread:1939 [sender.py:send():382] send: history 2024-11-13 19:18:53,041 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: summary_record 2024-11-13 19:18:53,041 INFO SenderThread:1939 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-11-13 19:18:53,186 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:18:53,187 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/wandb-summary.json 2024-11-13 19:18:54,187 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:18:55,187 INFO Thread-12 :1939 [dir_watcher.py:_on_file_modified():288] file/dir modified: /opt/ml/model//sft-codecontests-qwen_ds-code-contests_model-Qwen2.5-Coder-7B-Instruct_sch-cosine_lr-1e-6_bs-64_acc-8_len-2048/wandb/run-20241113_190246-kushalarora-rvv-main-2024-11-13-17-55-42-664-g0ro8r-algo-1/files/output.log 2024-11-13 19:18:55,768 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:18:55,922 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:18:55,922 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:18:57,159 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:19:02,160 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:19:07,160 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:19:10,768 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:19:10,923 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:19:10,923 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:19:13,077 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:19:17,045 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:19:19,046 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:19:24,047 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:19:25,768 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:19:25,923 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:19:25,923 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:19:29,078 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:19:34,079 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:19:39,079 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:19:40,768 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:19:40,922 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:19:40,923 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:19:45,078 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:19:47,048 DEBUG SenderThread:1939 [sender.py:send():382] send: stats 2024-11-13 19:19:51,050 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:19:55,768 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: internal_messages 2024-11-13 19:19:55,923 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: stop_status 2024-11-13 19:19:55,923 DEBUG SenderThread:1939 [sender.py:send_request():409] send_request: stop_status 2024-11-13 19:19:56,077 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:20:01,078 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report 2024-11-13 19:20:06,079 DEBUG HandlerThread:1939 [handler.py:handle_request():146] handle_request: status_report