{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/models/issues/2836","repository_url":"https://api.github.com/repos/tensorflow/models","labels_url":"https://api.github.com/repos/tensorflow/models/issues/2836/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/models/issues/2836/comments","events_url":"https://api.github.com/repos/tensorflow/models/issues/2836/events","html_url":"https://github.com/tensorflow/models/issues/2836","id":275219883,"number":2836,"title":"slim script doesn't work properly with Tensorboard when training on multiple GPUs","user":{"login":"qysnn","id":27019112,"avatar_url":"https://avatars1.githubusercontent.com/u/27019112?v=4","gravatar_id":"","url":"https://api.github.com/users/qysnn","html_url":"https://github.com/qysnn","followers_url":"https://api.github.com/users/qysnn/followers","following_url":"https://api.github.com/users/qysnn/following{/other_user}","gists_url":"https://api.github.com/users/qysnn/gists{/gist_id}","starred_url":"https://api.github.com/users/qysnn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qysnn/subscriptions","organizations_url":"https://api.github.com/users/qysnn/orgs","repos_url":"https://api.github.com/users/qysnn/repos","events_url":"https://api.github.com/users/qysnn/events{/privacy}","received_events_url":"https://api.github.com/users/qysnn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"sguada","id":1766524,"avatar_url":"https://avatars3.githubusercontent.com/u/1766524?v=4","gravatar_id":"","url":"https://api.github.com/users/sguada","html_url":"https://github.com/sguada","followers_url":"https://api.github.com/users/sguada/followers","following_url":"https://api.github.com/users/sguada/following{/other_user}","gists_url":"https://api.github.com/users/sguada/gists{/gist_id}","starred_url":"https://api.github.com/users/sguada/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sguada/subscriptions","organizations_url":"https://api.github.com/users/sguada/orgs","repos_url":"https://api.github.com/users/sguada/repos","events_url":"https://api.github.com/users/sguada/events{/privacy}","received_events_url":"https://api.github.com/users/sguada/received_events","type":"User","site_admin":false},"assignees":[{"login":"sguada","id":1766524,"avatar_url":"https://avatars3.githubusercontent.com/u/1766524?v=4","gravatar_id":"","url":"https://api.github.com/users/sguada","html_url":"https://github.com/sguada","followers_url":"https://api.github.com/users/sguada/followers","following_url":"https://api.github.com/users/sguada/following{/other_user}","gists_url":"https://api.github.com/users/sguada/gists{/gist_id}","starred_url":"https://api.github.com/users/sguada/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sguada/subscriptions","organizations_url":"https://api.github.com/users/sguada/orgs","repos_url":"https://api.github.com/users/sguada/repos","events_url":"https://api.github.com/users/sguada/events{/privacy}","received_events_url":"https://api.github.com/users/sguada/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2017-11-20T02:29:13Z","updated_at":"2018-04-25T20:42:33Z","closed_at":null,"author_association":"NONE","body":"### System information\r\n- **What is the top-level directory of the model you are using**:research/slim\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:Linux Ubuntu 16.04\r\n- **TensorFlow version (use command below)**:1.4\r\n- **GPU model and memory**:GTX 1080 X4\r\n- **Exact command to reproduce**:tensorboard --logdir=${TRAIN_DIR}\r\n\r\n\r\n### Describe the problem\r\nI am trying to use train_image_classifier.py to train Inception V4 model from scratch on flowers dataset. The training runs smoothly, but when I try to visualize it with TensorBoard as instructed in slim/README I find that TensorBoard will only show one step. Based on the output from TensorBoard, I think the problem should be multiple GPU instances write to the same TRAIN_DIR.\r\n\r\n### Source code / logs\r\n\r\n```\r\n...\r\nWARNING:tensorflow:Found more than one metagraph event per run. Overwriting the metagraph with the newest event.\r\nWARNING:tensorflow:Detected out of order event.step likely caused by a TensorFlow restart. Purging expired events from Tensorboard display between the previous step: -1 (timestamp: -1) and current step: 0 (timestamp: 1511141420.05). Removing 38 scalars, 635 histograms, 635 compressed histograms, 4 images, and 0 audio.\r\nWARNING:tensorflow:Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\r\nWARNING:tensorflow:Found more than one metagraph event per run. Overwriting the metagraph with the newest event.\r\nWARNING:tensorflow:Detected out of order event.step likely caused by a TensorFlow restart. Purging expired events from Tensorboard display between the previous step: -1 (timestamp: -1) and current step: 0 (timestamp: 1511141530.01). Removing 1 scalars, 0 histograms, 0 compressed histograms, 0 images, and 0 audio.\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/models/issues/comments/384427364","html_url":"https://github.com/tensorflow/models/issues/2836#issuecomment-384427364","issue_url":"https://api.github.com/repos/tensorflow/models/issues/2836","id":384427364,"user":{"login":"jichunshen","id":36704415,"avatar_url":"https://avatars2.githubusercontent.com/u/36704415?v=4","gravatar_id":"","url":"https://api.github.com/users/jichunshen","html_url":"https://github.com/jichunshen","followers_url":"https://api.github.com/users/jichunshen/followers","following_url":"https://api.github.com/users/jichunshen/following{/other_user}","gists_url":"https://api.github.com/users/jichunshen/gists{/gist_id}","starred_url":"https://api.github.com/users/jichunshen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jichunshen/subscriptions","organizations_url":"https://api.github.com/users/jichunshen/orgs","repos_url":"https://api.github.com/users/jichunshen/repos","events_url":"https://api.github.com/users/jichunshen/events{/privacy}","received_events_url":"https://api.github.com/users/jichunshen/received_events","type":"User","site_admin":false},"created_at":"2018-04-25T20:42:33Z","updated_at":"2018-04-25T20:42:33Z","author_association":"NONE","body":"What do you mean by traindir? The logdir or something else?"}}
{"actor":{"display_login":"jichunshen"}}