{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20750","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20750/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20750/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20750/events","html_url":"https://github.com/tensorflow/tensorflow/issues/20750","id":340769706,"node_id":"MDU6SXNzdWUzNDA3Njk3MDY=","number":20750,"title":"RunOptions.FULL_TRACE produce wrong timestamps","user":{"login":"ppwwyyxx","id":1381301,"node_id":"MDQ6VXNlcjEzODEzMDE=","avatar_url":"https://avatars3.githubusercontent.com/u/1381301?v=4","gravatar_id":"","url":"https://api.github.com/users/ppwwyyxx","html_url":"https://github.com/ppwwyyxx","followers_url":"https://api.github.com/users/ppwwyyxx/followers","following_url":"https://api.github.com/users/ppwwyyxx/following{/other_user}","gists_url":"https://api.github.com/users/ppwwyyxx/gists{/gist_id}","starred_url":"https://api.github.com/users/ppwwyyxx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ppwwyyxx/subscriptions","organizations_url":"https://api.github.com/users/ppwwyyxx/orgs","repos_url":"https://api.github.com/users/ppwwyyxx/repos","events_url":"https://api.github.com/users/ppwwyyxx/events{/privacy}","received_events_url":"https://api.github.com/users/ppwwyyxx/received_events","type":"User","site_admin":false},"labels":[{"id":404586594,"node_id":"MDU6TGFiZWw0MDQ1ODY1OTQ=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"b60205","default":false},{"id":473172988,"node_id":"MDU6TGFiZWw0NzMxNzI5ODg=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:bug/performance","name":"type:bug/performance","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":{"login":"zheng-xq","id":15736910,"node_id":"MDQ6VXNlcjE1NzM2OTEw","avatar_url":"https://avatars0.githubusercontent.com/u/15736910?v=4","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false},"assignees":[{"login":"zheng-xq","id":15736910,"node_id":"MDQ6VXNlcjE1NzM2OTEw","avatar_url":"https://avatars0.githubusercontent.com/u/15736910?v=4","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2018-07-12T19:33:31Z","updated_at":"2018-09-20T12:42:46Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**:yes\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:Ubuntu 16.04\r\n- **TensorFlow installed from (source or binary)**:binary\r\n- **TensorFlow version (use command below)**:v1.9.0-0-g25c197e023 1.9.0\r\n- **Python version**: 3.6\r\n- **Bazel version (if compiling from source)**:n/a\r\n- **GCC/Compiler version (if compiling from source)**:n/a\r\n- **CUDA/cuDNN version**:9.0.176/7.1.1\r\n- **GPU model and memory**:P100 16G (can reproduce on other GPU model as well)\r\n- **Exact command to reproduce**: see below\r\n\r\n```python\r\n#!/usr/bin/env python\r\n# -*- coding: utf-8 -*-\r\n\r\nimport sys\r\nimport os\r\nimport tqdm\r\nfrom datetime import datetime\r\nimport tensorflow as tf\r\nfrom tensorflow.python.client import timeline\r\n\r\ndef build():\r\n    image = tf.random_normal(shape=[64,32,32,3], dtype=tf.float32)\r\n    label = tf.random_uniform(shape=[64], maxval=10, dtype=tf.int32)\r\n    l = tf.transpose(image, [0, 3, 1, 2])\r\n    for k in range(1, 100):\r\n        l = tf.layers.conv2d(l, 16, 3, data_format='channels_first',\r\n                name='conv{}'.format(k), padding='SAME')\r\n    l = tf.reduce_mean(l, [2, 3])\r\n\r\n    logits = tf.layers.dense(l, 10, name='linear')\r\n    cost = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=label)\r\n    cost = tf.reduce_mean(cost, name='cross_entropy_loss')\r\n    return cost\r\n\r\nif __name__ == '__main__':\r\n    with tf.device('/gpu:0'):\r\n        cost1 = build()\r\n    with tf.device('/gpu:1'), tf.variable_scope(tf.get_variable_scope(), reuse=True):\r\n        cost2 = build()\r\n    cost = cost1 + cost2\r\n    opt = tf.train.GradientDescentOptimizer(0.1)\r\n    train_op = opt.minimize(cost)\r\n\r\n    def write_tracing(idx, metadata):\r\n        tl = timeline.Timeline(step_stats=metadata.step_stats)\r\n        fname = os.path.join(\r\n            '.', 'chrome-trace-{}.json'.format(idx))\r\n        with open(fname, 'w') as f:\r\n            f.write(tl.generate_chrome_trace_format(\r\n                show_dataflow=True, show_memory=True))\r\n\r\n    config = tf.ConfigProto()\r\n    config.allow_soft_placement = True\r\n    with tf.Session(config=config) as sess:\r\n        sess.run(tf.global_variables_initializer())\r\n        opt = tf.RunOptions()\r\n        opt.trace_level = tf.RunOptions.FULL_TRACE\r\n\r\n        for k in tqdm.trange(100):\r\n            meta = tf.RunMetadata()\r\n            sess.run(train_op, options=opt, run_metadata=meta)\r\n\r\n            write_tracing(k, meta)\r\n\r\n            for devst in meta.step_stats.dev_stats:\r\n                for ns in devst.node_stats:\r\n                    micro = timestamp = ns.all_start_micros // 1000000\r\n                    timestamp = datetime.fromtimestamp(timestamp)\r\n                    diff = timestamp - datetime.now()\r\n                    if diff.days > 100:\r\n                        print(k, micro, timestamp)\r\n                        #import IPython as IP; IP.embed()\r\n                        #sys.exit()\r\n```\r\n\r\nThe code above trains a CNN on two GPUs with `FULL_TRACE` enabled. The returned profiling information contains correct timestamps, but sometimes contains timestamps that are many years in the future. It prints the following output:\r\n```\r\n...\r\n13 18446744073 2554-07-21 16:34:33                                                                                                                                                                  \r\n13 18446744073 2554-07-21 16:34:33               \r\n13 18446744073 2554-07-21 16:34:33                                                                                                                                                                  \r\n13 18446744073 2554-07-21 16:34:33               \r\n13 18446744073 2554-07-21 16:34:33               \r\n13 18446744073 2554-07-21 16:34:33                                                                                                                                                                  \r\n13 18446744073 2554-07-21 16:34:33               \r\n13 18446744073 2554-07-21 16:34:33                                                                                                                                                                  \r\n13 18446744073 2554-07-21 16:34:33               \r\n13 18446744073 2554-07-21 16:34:33               \r\n13 18446744073 2554-07-21 16:34:33\r\n...\r\n```\r\n\r\nThe issue was originally reported at https://github.com/tensorpack/tensorpack/issues/819.\r\nThe issue is more likely to happen after running about 50 steps.\r\nThe issue seems to disappear when training on one GPU, or training a small model."},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/423168696","html_url":"https://github.com/tensorflow/tensorflow/issues/20750#issuecomment-423168696","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20750","id":423168696,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzE2ODY5Ng==","user":{"login":"tensorflowbutler","id":28546240,"node_id":"MDQ6VXNlcjI4NTQ2MjQw","avatar_url":"https://avatars2.githubusercontent.com/u/28546240?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflowbutler","html_url":"https://github.com/tensorflowbutler","followers_url":"https://api.github.com/users/tensorflowbutler/followers","following_url":"https://api.github.com/users/tensorflowbutler/following{/other_user}","gists_url":"https://api.github.com/users/tensorflowbutler/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflowbutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflowbutler/subscriptions","organizations_url":"https://api.github.com/users/tensorflowbutler/orgs","repos_url":"https://api.github.com/users/tensorflowbutler/repos","events_url":"https://api.github.com/users/tensorflowbutler/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflowbutler/received_events","type":"User","site_admin":false},"created_at":"2018-09-20T12:42:46Z","updated_at":"2018-09-20T12:42:46Z","author_association":"MEMBER","body":"Nagging Assignee @zheng-xq: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly."}}
{"actor":{"display_login":"tensorflowbutler"}}