{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21492","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21492/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21492/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21492/events","html_url":"https://github.com/tensorflow/tensorflow/issues/21492","id":348904600,"node_id":"MDU6SXNzdWUzNDg5MDQ2MDA=","number":21492,"title":"Public API to preempt tf.train.Server","user":{"login":"superbobry","id":185856,"node_id":"MDQ6VXNlcjE4NTg1Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/185856?v=4","gravatar_id":"","url":"https://api.github.com/users/superbobry","html_url":"https://github.com/superbobry","followers_url":"https://api.github.com/users/superbobry/followers","following_url":"https://api.github.com/users/superbobry/following{/other_user}","gists_url":"https://api.github.com/users/superbobry/gists{/gist_id}","starred_url":"https://api.github.com/users/superbobry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/superbobry/subscriptions","organizations_url":"https://api.github.com/users/superbobry/orgs","repos_url":"https://api.github.com/users/superbobry/repos","events_url":"https://api.github.com/users/superbobry/events{/privacy}","received_events_url":"https://api.github.com/users/superbobry/received_events","type":"User","site_admin":false},"labels":[{"id":404586594,"node_id":"MDU6TGFiZWw0MDQ1ODY1OTQ=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"b60205","default":false}],"state":"open","locked":false,"assignee":{"login":"karmel","id":667809,"node_id":"MDQ6VXNlcjY2NzgwOQ==","avatar_url":"https://avatars1.githubusercontent.com/u/667809?v=4","gravatar_id":"","url":"https://api.github.com/users/karmel","html_url":"https://github.com/karmel","followers_url":"https://api.github.com/users/karmel/followers","following_url":"https://api.github.com/users/karmel/following{/other_user}","gists_url":"https://api.github.com/users/karmel/gists{/gist_id}","starred_url":"https://api.github.com/users/karmel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karmel/subscriptions","organizations_url":"https://api.github.com/users/karmel/orgs","repos_url":"https://api.github.com/users/karmel/repos","events_url":"https://api.github.com/users/karmel/events{/privacy}","received_events_url":"https://api.github.com/users/karmel/received_events","type":"User","site_admin":false},"assignees":[{"login":"karmel","id":667809,"node_id":"MDQ6VXNlcjY2NzgwOQ==","avatar_url":"https://avatars1.githubusercontent.com/u/667809?v=4","gravatar_id":"","url":"https://api.github.com/users/karmel","html_url":"https://github.com/karmel","followers_url":"https://api.github.com/users/karmel/followers","following_url":"https://api.github.com/users/karmel/following{/other_user}","gists_url":"https://api.github.com/users/karmel/gists{/gist_id}","starred_url":"https://api.github.com/users/karmel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karmel/subscriptions","organizations_url":"https://api.github.com/users/karmel/orgs","repos_url":"https://api.github.com/users/karmel/repos","events_url":"https://api.github.com/users/karmel/events{/privacy}","received_events_url":"https://api.github.com/users/karmel/received_events","type":"User","site_admin":false}],"milestone":null,"comments":12,"created_at":"2018-08-08T21:49:12Z","updated_at":"2018-09-29T18:41:43Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: N/A\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: N/A\r\n- **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device**: N/A\r\n- **TensorFlow installed from (source or binary)**: binary\r\n- **TensorFlow version (use command below)**: v1.9.0-0-g25c197e023 1.9.0\r\n- **Python version**: N/A\r\n- **Bazel version (if compiling from source)**: N/A\r\n- **GCC/Compiler version (if compiling from source)**:\r\n- **CUDA/cuDNN version**: N/A\r\n- **GPU model and memory**: N/A\r\n- **Exact command to reproduce**: N/A\r\n\r\n### Describe the problem\r\n\r\n`tf.train_and_evaluate` uses `time.sleep` to (optimistically) synchronize the startup of chief/worker nodes in the distributed mode (see [estimator/training.py](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/estimator/training.py#L753)). The implicit assumption in this logic is that by the time the worker is spawned, the chief has already started its `tf.train.Server`, i.e. the scheduler should be aware of the assumption and should schedule and initialize the chief first. This might not be easily achievable on general purpose systems like YARN.\r\n\r\nOne possible solution to this is to synchronize the chief/worker tasks on a barrier, and then preemptively start the server right after the barrier, but prior to calling `train_and_evaluate`. This does not eliminate the race condition entirely but makes it much less likely in practice. The only problem here is that `train_and_evaluate` does not provide a documented way to account for preempted servers. The undocumented way is:\r\n\r\nhttps://github.com/tensorflow/tensorflow/blob/2b4fd1c2b7a37367e61bbae3d27d194a894cb7bb/tensorflow/python/estimator/training.py#L747-L748\r\n\r\nI was wondering if it would be possible to make this part of the `train_and_evaluate` contract public or, alternatively, address the issue in another way?"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/425666939","html_url":"https://github.com/tensorflow/tensorflow/issues/21492#issuecomment-425666939","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21492","id":425666939,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNTY2NjkzOQ==","user":{"login":"tensorflowbutler","id":28546240,"node_id":"MDQ6VXNlcjI4NTQ2MjQw","avatar_url":"https://avatars2.githubusercontent.com/u/28546240?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflowbutler","html_url":"https://github.com/tensorflowbutler","followers_url":"https://api.github.com/users/tensorflowbutler/followers","following_url":"https://api.github.com/users/tensorflowbutler/following{/other_user}","gists_url":"https://api.github.com/users/tensorflowbutler/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflowbutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflowbutler/subscriptions","organizations_url":"https://api.github.com/users/tensorflowbutler/orgs","repos_url":"https://api.github.com/users/tensorflowbutler/repos","events_url":"https://api.github.com/users/tensorflowbutler/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflowbutler/received_events","type":"User","site_admin":false},"created_at":"2018-09-29T18:41:43Z","updated_at":"2018-09-29T18:41:43Z","author_association":"MEMBER","body":"Nagging Assignee @karmel: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly."}}
{"actor":{"display_login":"tensorflowbutler"}}