{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2461","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2461/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2461/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2461/events","html_url":"https://github.com/tensorflow/tensorflow/issues/2461","id":156132575,"number":2461,"title":"Server restarts silently on kubernetes and variables reset after restart","user":{"login":"perhapszzy","id":7953637,"avatar_url":"https://avatars.githubusercontent.com/u/7953637?v=3","gravatar_id":"","url":"https://api.github.com/users/perhapszzy","html_url":"https://github.com/perhapszzy","followers_url":"https://api.github.com/users/perhapszzy/followers","following_url":"https://api.github.com/users/perhapszzy/following{/other_user}","gists_url":"https://api.github.com/users/perhapszzy/gists{/gist_id}","starred_url":"https://api.github.com/users/perhapszzy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/perhapszzy/subscriptions","organizations_url":"https://api.github.com/users/perhapszzy/orgs","repos_url":"https://api.github.com/users/perhapszzy/repos","events_url":"https://api.github.com/users/perhapszzy/events{/privacy}","received_events_url":"https://api.github.com/users/perhapszzy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-05-22T03:20:00Z","updated_at":"2016-05-22T19:37:42Z","closed_at":null,"body":"### Environment info\r\nOperating System:\r\nKubernetes with default images\r\n\r\n### Steps to reproduce\r\nHard to reproduce due to lack of log\r\n\r\n### Logs or other output that would be helpful\r\nThe kubernetes status:\r\n```\r\nNAME               READY     STATUS    RESTARTS   AGE\r\ntf-ps2-l2ka8       1/1       Running   0          3h\r\ntf-ps4-2jmuf       1/1       Running   1          3h\r\ntf-worker1-fm1t8   1/1       Running   1          3h\r\ntf-worker3-ujd7h   1/1       Running   0          3h\r\ntf-worker5-ymx71   1/1       Running   0          3h\r\n```\r\n\r\nThe log regarding restarted parameter server:\r\n```\r\nzeyu@perhaps:~$ kubectl logs -p tf-ps4-2jmuf\r\nI tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job worker -> {tf-worker1:2222, tf-worker3:2222, tf-worker5:2222}\r\nI tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job ps -> {tf-ps2:2222, localhost:2222}\r\nI tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:202] Started server with target: grpc://localhost:2222\r\n```\r\nThe log of current running server: (almost the same)\r\n```\r\nzeyu@perhaps:~$ kubectl logs  tf-ps4-2jmuf\r\nI tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job worker -> {tf-worker1:2222, tf-worker3:2222, tf-worker5:2222}\r\nI tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:206] Initialize HostPortsGrpcChannelCache for job ps -> {tf-ps2:2222, localhost:2222}\r\nI tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:202] Started server with target: grpc://localhost:2222\r\n```\r\n\r\nAs the result of the restart, the global count  reset to 0 and the running log looks like:\r\n```\r\nTraining step 269, global step 802\r\nTraining step 270, global step 805\r\nTraining step 271, global step 808\r\nTraining step 272, global step 810\r\nTraining step 273, global step 3\r\nTraining step 274, global step 6\r\nTraining step 275, global step 12\r\nTraining step 276, global step 19\r\nTraining step 277, global step 22\r\nTraining step 278, global step 27\r\nTraining step 279, global step 32\r\n```\r\n\r\nIs there any way that I can materialize the variables (like hostpath or persistent disk?) so that they won't be affected by restarts?\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/220851537","html_url":"https://github.com/tensorflow/tensorflow/issues/2461#issuecomment-220851537","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2461","id":220851537,"user":{"login":"perhapszzy","id":7953637,"avatar_url":"https://avatars.githubusercontent.com/u/7953637?v=3","gravatar_id":"","url":"https://api.github.com/users/perhapszzy","html_url":"https://github.com/perhapszzy","followers_url":"https://api.github.com/users/perhapszzy/followers","following_url":"https://api.github.com/users/perhapszzy/following{/other_user}","gists_url":"https://api.github.com/users/perhapszzy/gists{/gist_id}","starred_url":"https://api.github.com/users/perhapszzy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/perhapszzy/subscriptions","organizations_url":"https://api.github.com/users/perhapszzy/orgs","repos_url":"https://api.github.com/users/perhapszzy/repos","events_url":"https://api.github.com/users/perhapszzy/events{/privacy}","received_events_url":"https://api.github.com/users/perhapszzy/received_events","type":"User","site_admin":false},"created_at":"2016-05-22T19:37:42Z","updated_at":"2016-05-22T19:37:42Z","body":"Yes, I did use the ```tf.train.Supervisor```, but the job may not be running for 600 seconds. Are the checkpoints stored on the server side? I didn't see any checkpoints stored on my local machine (where I did see the summary files) without explicitly call ```sv.saver.save```"}}
{"actor":{"display_login":"perhapszzy"}}