{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla/DeepSpeech/issues/704","repository_url":"https://api.github.com/repos/mozilla/DeepSpeech","labels_url":"https://api.github.com/repos/mozilla/DeepSpeech/issues/704/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/DeepSpeech/issues/704/comments","events_url":"https://api.github.com/repos/mozilla/DeepSpeech/issues/704/events","html_url":"https://github.com/mozilla/DeepSpeech/issues/704","id":241513005,"node_id":"MDU6SXNzdWUyNDE1MTMwMDU=","number":704,"title":"Cluster coordinator hangs at the end of training","user":{"login":"pecastro","id":352625,"node_id":"MDQ6VXNlcjM1MjYyNQ==","avatar_url":"https://avatars3.githubusercontent.com/u/352625?v=4","gravatar_id":"","url":"https://api.github.com/users/pecastro","html_url":"https://github.com/pecastro","followers_url":"https://api.github.com/users/pecastro/followers","following_url":"https://api.github.com/users/pecastro/following{/other_user}","gists_url":"https://api.github.com/users/pecastro/gists{/gist_id}","starred_url":"https://api.github.com/users/pecastro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pecastro/subscriptions","organizations_url":"https://api.github.com/users/pecastro/orgs","repos_url":"https://api.github.com/users/pecastro/repos","events_url":"https://api.github.com/users/pecastro/events{/privacy}","received_events_url":"https://api.github.com/users/pecastro/received_events","type":"User","site_admin":false},"labels":[{"id":386594038,"node_id":"MDU6TGFiZWwzODY1OTQwMzg=","url":"https://api.github.com/repos/mozilla/DeepSpeech/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":{"login":"tilmankamp","id":5991088,"node_id":"MDQ6VXNlcjU5OTEwODg=","avatar_url":"https://avatars2.githubusercontent.com/u/5991088?v=4","gravatar_id":"","url":"https://api.github.com/users/tilmankamp","html_url":"https://github.com/tilmankamp","followers_url":"https://api.github.com/users/tilmankamp/followers","following_url":"https://api.github.com/users/tilmankamp/following{/other_user}","gists_url":"https://api.github.com/users/tilmankamp/gists{/gist_id}","starred_url":"https://api.github.com/users/tilmankamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tilmankamp/subscriptions","organizations_url":"https://api.github.com/users/tilmankamp/orgs","repos_url":"https://api.github.com/users/tilmankamp/repos","events_url":"https://api.github.com/users/tilmankamp/events{/privacy}","received_events_url":"https://api.github.com/users/tilmankamp/received_events","type":"User","site_admin":false},"assignees":[{"login":"tilmankamp","id":5991088,"node_id":"MDQ6VXNlcjU5OTEwODg=","avatar_url":"https://avatars2.githubusercontent.com/u/5991088?v=4","gravatar_id":"","url":"https://api.github.com/users/tilmankamp","html_url":"https://github.com/tilmankamp","followers_url":"https://api.github.com/users/tilmankamp/followers","following_url":"https://api.github.com/users/tilmankamp/following{/other_user}","gists_url":"https://api.github.com/users/tilmankamp/gists{/gist_id}","starred_url":"https://api.github.com/users/tilmankamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tilmankamp/subscriptions","organizations_url":"https://api.github.com/users/tilmankamp/orgs","repos_url":"https://api.github.com/users/tilmankamp/repos","events_url":"https://api.github.com/users/tilmankamp/events{/privacy}","received_events_url":"https://api.github.com/users/tilmankamp/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2017-07-09T09:35:57Z","updated_at":"2018-07-11T12:29:07Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"This does not happen in a clusterless scenario.\r\nBut in a simple 2 node (PS + Worker) setup or bigger the Coordinator will always hang at the end of training.\r\n\r\nPS:\r\n```\r\nbin/run-ldc93s1.sh      --ps_hosts=host0:2233      --worker_hosts=host0:2222      --job_name=ps --task_index=0 --log_level 0 |& tee -a ps_train.log\r\n```\r\nPS Output:\r\n```\r\n+ '[' '!' -f DeepSpeech.py ']'\r\n+ '[' '!' -f data/ldc93s1/ldc93s1.csv ']'\r\n+ '[' -d data/ldc93s1/test/ps ']'\r\n+ checkpoint_dir=data/ldc93s1/test/ps\r\n+ python3 -u DeepSpeech.py --train_files data/ldc93s1/ldc93s1.csv --dev_files data/ldc93s1/ldc93s1.csv --test_files data/ldc93s1/ldc93s1.csv --train_batch_size 1 --dev_batch_size 1 --test_batch_size 1 --n_hidden 494 --epoch 2 --checkpoint_dir data/ldc93s1/test/ps --export_dir data/ldc93s1/test/ps/export --fulltrace True --display_step 1 --log_level 0 --ps_hosts=host0:2233 --worker_hosts=host0:2222 --job_name=ps --task_index=0 --log_level 0\r\n2017-07-09 09:12:43.287531: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.\r\n2017-07-09 09:12:43.287572: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.\r\n2017-07-09 09:12:43.301837: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> localhost:2233}\r\n2017-07-09 09:12:43.301879: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> host0:2222}\r\n2017-07-09 09:12:43.302765: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:2233\r\nD Waiting for stop token...\r\n2017-07-09 09:12:45.991076: I tensorflow/core/distributed_runtime/master_session.cc:995] Start master session 32dce54fc71a4d98 with config:\r\n\r\nD Got a stop token from worker 0.\r\nD Session closed.\r\nD Server stopped.\r\n```\r\n\r\nWorker:\r\n```\r\nbin/run-ldc93s1.sh      --ps_hosts=host0:2233      --worker_hosts=host0:2222      --job_name=worker --task_index=0 --log_traffic True --coord_host host0 --coord_port 2501 --log_level 0 |& tee -a worker0_train.log\r\n```\r\n\r\nWorker Output:\r\n```\r\n+ '[' '!' -f DeepSpeech.py ']'\r\n+ '[' '!' -f data/ldc93s1/ldc93s1.csv ']'\r\n+ '[' -d data/ldc93s1/test/worker0 ']'\r\n+ checkpoint_dir=data/ldc93s1/test/worker0\r\n+ python3 -u DeepSpeech.py --train_files data/ldc93s1/ldc93s1.csv --dev_files data/ldc93s1/ldc93s1.csv --test_files data/ldc93s1/ldc93s1.csv --train_batch_size 1 --dev_batch_size 1 --test_batch_size 1 --n_hidden 494 --epoch 2 --checkpoint_dir data/ldc93s1/test/worker0 --export_dir data/ldc93s1/test/worker0/export --fulltrace True --display_step 1 --log_level 0 --ps_hosts=host0:2233 --worker_hosts=host0:2222 --job_name=worker --task_index=0 --log_traffic True --coord_host host0 --coord_port 2501 --log_level 0\r\nD Starting coordinator...\r\nD Coordinator started.\r\n2017-07-09 09:12:45.360117: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.\r\n2017-07-09 09:12:45.360147: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.\r\n2017-07-09 09:12:45.375372: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> host0:2233}\r\n2017-07-09 09:12:45.375431: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> localhost:2222}\r\n2017-07-09 09:12:45.376382: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:316] Started server with target: grpc://localhost:2222\r\nW Parameter --validation_step needs to be >0 for early stopping to work\r\n2017-07-09 09:12:47.869413: I tensorflow/core/distributed_runtime/master_session.cc:995] Start master session 95bc524f41cc8c0b with config:\r\nallow_soft_placement: true\r\n\r\nD Starting queue runners...\r\n------------------------------------------------------------------------\r\nWARNING: libdeepspeech failed to load, resorting to deprecated code\r\n         Refer to README.md for instructions on installing libdeepspeech\r\n------------------------------------------------------------------------\r\nD Queue runners started.\r\nI STARTING Optimization\r\nD step: 0\r\nD epoch: 0\r\nD target epoch: 2\r\nD steps per epoch: 1\r\nD number of batches in train set: 1\r\nD batches per job: 1\r\nD batches per step: 1\r\nD number of jobs in train set: 1\r\nD number of jobs already trained in first epoch: 0\r\nD Got new Job (ID: 2, worker: 0, epoch: 0, set_name: train)\r\nD Computing Job (ID: 2, worker: 0, epoch: 0, set_name: train)...\r\nD Starting batch...\r\nD Finished batch step 0.\r\nD Sending Job (ID: 2, worker: 0, epoch: 0, set_name: train)...\r\nD Training of Epoch 0 - Moved Job (ID: 2, worker: 0, epoch: 0, set_name: train) from running to done.\r\nI Training of Epoch 0 - WER: 2.454545, loss: 376.378387451, mean edit distance: 1.269231\r\nI --------------------------------------------------------------------------------\r\nI WER: 2.454545, loss: 376.378387, mean edit distance: 1.269231\r\nI  - src: \"she had your dark suit in greasy wash water all year\"\r\nI  - res: \"and deep p p m i p p p d d m p m m m p i p i d philip p p p p pypypn\"\r\nI --------------------------------------------------------------------------------\r\nD Computing Job (ID: 4, worker: 0, epoch: 1, set_name: train)...\r\nD Starting batch...\r\nD Finished batch step 1.\r\nD Sending Job (ID: 4, worker: 0, epoch: 1, set_name: train)...\r\nD Training of Epoch 1 - Moved Job (ID: 4, worker: 0, epoch: 1, set_name: train) from running to done.\r\nI Training of Epoch 1 - WER: 1.000000, loss: 203.96522522, mean edit distance: 0.961538\r\nI --------------------------------------------------------------------------------\r\nI WER: 1.000000, loss: 203.965225, mean edit distance: 0.961538\r\nI  - src: \"she had your dark suit in greasy wash water all year\"\r\nI  - res: \"\"\r\nI --------------------------------------------------------------------------------\r\nI FINISHED Optimization - training time: 0:00:07\r\nD Computing Job (ID: 6, worker: 0, epoch: 2, set_name: test)...\r\nD Starting batch...\r\nD Finished batch step 1.\r\nD Sending Job (ID: 6, worker: 0, epoch: 2, set_name: test)...\r\nD Test of Epoch 2 - Moved Job (ID: 6, worker: 0, epoch: 2, set_name: test) from running to done.\r\nI Test of Epoch 2 - WER: 1.000000, loss: 203.1690979, mean edit distance: 0.961538\r\nI --------------------------------------------------------------------------------\r\nI WER: 1.000000, loss: 203.169098, mean edit distance: 0.961538\r\nI  - src: \"she had your dark suit in greasy wash water all year\"\r\nI  - res: \"\"\r\nI --------------------------------------------------------------------------------\r\nD No jobs left for worker 0.\r\nD Epochs - running: 0, done: 3\r\nD Closing queues...\r\nD Queues closed.\r\nD Sending stop token to ps 0...\r\nD Sent stop token to ps 0.\r\n```\r\n\r\nWorker will be hanging in this line https://github.com/mozilla/DeepSpeech/blob/3ac527cecc6b31f0aa86a82e84174f77cbed66cc/DeepSpeech.py#L1508 apparently still trying to communicate with a PS which is gone.\r\n\r\nUsing Tensorflow v1.2.0 but the same also happens with v1.1.0.\r\nUsing Python3 but the same also happens with Python2.\r\n"},"comment":{"url":"https://api.github.com/repos/mozilla/DeepSpeech/issues/comments/404151481","html_url":"https://github.com/mozilla/DeepSpeech/issues/704#issuecomment-404151481","issue_url":"https://api.github.com/repos/mozilla/DeepSpeech/issues/704","id":404151481,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNDE1MTQ4MQ==","user":{"login":"yash-sam","id":41115129,"node_id":"MDQ6VXNlcjQxMTE1MTI5","avatar_url":"https://avatars0.githubusercontent.com/u/41115129?v=4","gravatar_id":"","url":"https://api.github.com/users/yash-sam","html_url":"https://github.com/yash-sam","followers_url":"https://api.github.com/users/yash-sam/followers","following_url":"https://api.github.com/users/yash-sam/following{/other_user}","gists_url":"https://api.github.com/users/yash-sam/gists{/gist_id}","starred_url":"https://api.github.com/users/yash-sam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yash-sam/subscriptions","organizations_url":"https://api.github.com/users/yash-sam/orgs","repos_url":"https://api.github.com/users/yash-sam/repos","events_url":"https://api.github.com/users/yash-sam/events{/privacy}","received_events_url":"https://api.github.com/users/yash-sam/received_events","type":"User","site_admin":false},"created_at":"2018-07-11T12:29:07Z","updated_at":"2018-07-11T12:29:07Z","author_association":"NONE","body":"Please @tilmankamp , @pecastro , do you guys have any news on this? Situation is exactly the same as described above, worker0 hangs at the end of the training, after \"Sending stop token to ps\". I had no success in finding a solution here or in mozilla discourse site. I'm using tensorflow 1.8.0, deepspeech 0.1.1, training goes fine in a single process. Worked hard to get this nice project running in a cluster with several machines, now I need to overcome this in order to proceed to training with real data in my own language. Thanks!"}}
{"actor":{"display_login":"yash-sam"}}