{"action":"created","issue":{"url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/359","repository_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark","labels_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/359/labels{/name}","comments_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/359/comments","events_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/359/events","html_url":"https://github.com/yahoo/TensorFlowOnSpark/issues/359","id":373873016,"node_id":"MDU6SXNzdWUzNzM4NzMwMTY=","number":359,"title":"Exception: timed out waiting for reservations to complete","user":{"login":"wanghongling123","id":35447111,"node_id":"MDQ6VXNlcjM1NDQ3MTEx","avatar_url":"https://avatars0.githubusercontent.com/u/35447111?v=4","gravatar_id":"","url":"https://api.github.com/users/wanghongling123","html_url":"https://github.com/wanghongling123","followers_url":"https://api.github.com/users/wanghongling123/followers","following_url":"https://api.github.com/users/wanghongling123/following{/other_user}","gists_url":"https://api.github.com/users/wanghongling123/gists{/gist_id}","starred_url":"https://api.github.com/users/wanghongling123/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wanghongling123/subscriptions","organizations_url":"https://api.github.com/users/wanghongling123/orgs","repos_url":"https://api.github.com/users/wanghongling123/repos","events_url":"https://api.github.com/users/wanghongling123/events{/privacy}","received_events_url":"https://api.github.com/users/wanghongling123/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-10-25T10:12:46Z","updated_at":"2018-10-26T05:39:59Z","closed_at":null,"author_association":"NONE","body":"Hi,I get an error when I am training.When I run Command :yarn logs -applicationId ...,I got results as follows:\r\nTraceback (most recent call last):\r\n  File \"mnist_spark.py\", line 69, in <module>\r\n    cluster = TFCluster.run(sc, mnist_dist.map_fun, args, args.cluster_size, num_ps, args.tensorboard, TFCluster.InputMode.SPARK, log_dir=args.model)\r\n  File \"/usr/local/hadoop-2.7.1/tmp/nm-local-dir/usercache/honglingwang/appcache/application_1540460145880_0005/container_1540460145880_0005_01_000001/tfspark.zip/tensorflowonspark/TFCluster.py\", line 297, in run\r\n  File \"/usr/local/hadoop-2.7.1/tmp/nm-local-dir/usercache/honglingwang/appcache/application_1540460145880_0005/container_1540460145880_0005_01_000001/tfspark.zip/tensorflowonspark/reservation.py\", line 121, in await_reservations\r\nException: timed out waiting for reservations to complete\r\nEnd of LogType:stdout\r\nBut How to proceed?\r\n"},"comment":{"url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/comments/433292538","html_url":"https://github.com/yahoo/TensorFlowOnSpark/issues/359#issuecomment-433292538","issue_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/359","id":433292538,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMzI5MjUzOA==","user":{"login":"wanghongling123","id":35447111,"node_id":"MDQ6VXNlcjM1NDQ3MTEx","avatar_url":"https://avatars0.githubusercontent.com/u/35447111?v=4","gravatar_id":"","url":"https://api.github.com/users/wanghongling123","html_url":"https://github.com/wanghongling123","followers_url":"https://api.github.com/users/wanghongling123/followers","following_url":"https://api.github.com/users/wanghongling123/following{/other_user}","gists_url":"https://api.github.com/users/wanghongling123/gists{/gist_id}","starred_url":"https://api.github.com/users/wanghongling123/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wanghongling123/subscriptions","organizations_url":"https://api.github.com/users/wanghongling123/orgs","repos_url":"https://api.github.com/users/wanghongling123/repos","events_url":"https://api.github.com/users/wanghongling123/events{/privacy}","received_events_url":"https://api.github.com/users/wanghongling123/received_events","type":"User","site_admin":false},"created_at":"2018-10-26T05:39:59Z","updated_at":"2018-10-26T05:39:59Z","author_association":"NONE","body":"I doubt this is the point\r\n  def await_reservations(self, sc, status={}, timeout=600):\r\n    \"\"\"Block until all reservations are received.\"\"\"\r\n    timespent = 0\r\n    while not self.reservations.done():\r\n      logging.info(\"waiting for {0} reservations\".format(self.reservations.remaining()))\r\n      # check status flags for any errors\r\n      if 'error' in status:\r\n        sc.cancelAllJobs()\r\n        sc.stop()\r\n        sys.exit(1)\r\n      time.sleep(1)\r\n      timespent += 1\r\n      if (timespent > timeout):\r\n        raise Exception(\"**timed out waiting for reservations to complete**\")\r\n    logging.info(\"all reservations completed\")\r\n    return self.reservations.get()\r\nI want to know why the parameter timeout=600 is introduced .Can I increase it? "}}
{"actor":{"display_login":"wanghongling123"}}