{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6416","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6416/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6416/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6416/events","html_url":"https://github.com/tensorflow/tensorflow/issues/6416","id":196625831,"number":6416,"title":"Tensorflow fault tolerance","user":{"login":"chaitanya2692","id":8254695,"avatar_url":"https://avatars.githubusercontent.com/u/8254695?v=3","gravatar_id":"","url":"https://api.github.com/users/chaitanya2692","html_url":"https://github.com/chaitanya2692","followers_url":"https://api.github.com/users/chaitanya2692/followers","following_url":"https://api.github.com/users/chaitanya2692/following{/other_user}","gists_url":"https://api.github.com/users/chaitanya2692/gists{/gist_id}","starred_url":"https://api.github.com/users/chaitanya2692/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chaitanya2692/subscriptions","organizations_url":"https://api.github.com/users/chaitanya2692/orgs","repos_url":"https://api.github.com/users/chaitanya2692/repos","events_url":"https://api.github.com/users/chaitanya2692/events{/privacy}","received_events_url":"https://api.github.com/users/chaitanya2692/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-12-20T09:36:41Z","updated_at":"2016-12-21T02:04:51Z","closed_at":"2016-12-21T01:27:21Z","body":"Hello,\r\n\r\nI have been using the following simple code to learn about the behavior of distributed tensorflow (which claims to be fault tolerant):-\r\n\r\n  ```\r\nimport tensorflow as tf\r\ncluster = tf.train.ClusterSpec({\"local\": [\"localhost:2222\", \"localhost:2223\",\"localhost:2224\", \"localhost:2225\"]})\r\nx = tf.constant(2)\r\nwith tf.device(\"/job:local/task:1\"):\r\n    y1 = x + 300\r\n with tf.device(\"/job:local/task:2\"):\r\n    y2 = x**2\r\nwith tf.device(\"/job:local/task:3\"):\r\n    y3 = 5*x\r\nwith tf.device(\"/job:local/task:0\"):\r\n    y0 = x - 66\r\n    y = y0 + y1 + y2 + y3\r\n\r\nmodel = tf.initialize_all_variables()\r\n\r\nwith tf.Session(\"grpc://localhost:2222\") as sess:\r\n    sess.run(model)\r\n    print(sess.run(y0))\r\n    print('\\n')\r\n    print(sess.run(y1))\r\n    print('\\n')\r\n    print(sess.run(y2))\r\n    print('\\n')\r\n    print(sess.run(y3))\r\n    print('\\n')\r\n    result = sess.run(y)\r\n    print(result)\r\n```\r\n\r\nI create workers by running a separate program as given below:-\r\n\r\n```\r\n#get task number from command line\r\nimport sys\r\ntask_number = int(sys.argv[1])\r\n\r\nimport tensorflow as tf\r\n\r\ncluster = tf.train.ClusterSpec({\"local\": [\"localhost:2222\", \"localhost:2223\", \"localhost:2224\", \"localhost:2225\"]})\r\nserver = tf.train.Server(cluster, job_name=\"local\", task_index=task_number)\r\n\r\nprint(\"Starting server #{}\".format(task_number))\r\n\r\nserver.start()\r\nserver.join()\r\n\r\n```\r\nBut when I stop one of the servers(machines), the whole program stops functioning instead of assigning the job to another machine. Is tensorflow fault tolerant when a machine goes down? "},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/268414095","html_url":"https://github.com/tensorflow/tensorflow/issues/6416#issuecomment-268414095","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6416","id":268414095,"user":{"login":"mrry","id":192142,"avatar_url":"https://avatars.githubusercontent.com/u/192142?v=3","gravatar_id":"","url":"https://api.github.com/users/mrry","html_url":"https://github.com/mrry","followers_url":"https://api.github.com/users/mrry/followers","following_url":"https://api.github.com/users/mrry/following{/other_user}","gists_url":"https://api.github.com/users/mrry/gists{/gist_id}","starred_url":"https://api.github.com/users/mrry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrry/subscriptions","organizations_url":"https://api.github.com/users/mrry/orgs","repos_url":"https://api.github.com/users/mrry/repos","events_url":"https://api.github.com/users/mrry/events{/privacy}","received_events_url":"https://api.github.com/users/mrry/received_events","type":"User","site_admin":false},"created_at":"2016-12-21T02:04:51Z","updated_at":"2016-12-21T02:04:51Z","body":"The session may be in an undefined state at that point, so YMMV if you try to keep using it. The various wrapper libraries will re-create the session in those cases."}}
{"actor":{"display_login":"mrry"}}