{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19062","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19062/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19062/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19062/events","html_url":"https://github.com/tensorflow/tensorflow/issues/19062","id":319991753,"node_id":"MDU6SXNzdWUzMTk5OTE3NTM=","number":19062,"title":"train_and_evaluate does not preserve the Dataset iterator state across train/eval","user":{"login":"superbobry","id":185856,"node_id":"MDQ6VXNlcjE4NTg1Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/185856?v=4","gravatar_id":"","url":"https://api.github.com/users/superbobry","html_url":"https://github.com/superbobry","followers_url":"https://api.github.com/users/superbobry/followers","following_url":"https://api.github.com/users/superbobry/following{/other_user}","gists_url":"https://api.github.com/users/superbobry/gists{/gist_id}","starred_url":"https://api.github.com/users/superbobry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/superbobry/subscriptions","organizations_url":"https://api.github.com/users/superbobry/orgs","repos_url":"https://api.github.com/users/superbobry/repos","events_url":"https://api.github.com/users/superbobry/events{/privacy}","received_events_url":"https://api.github.com/users/superbobry/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"ispirmustafa","id":19293677,"node_id":"MDQ6VXNlcjE5MjkzNjc3","avatar_url":"https://avatars1.githubusercontent.com/u/19293677?v=4","gravatar_id":"","url":"https://api.github.com/users/ispirmustafa","html_url":"https://github.com/ispirmustafa","followers_url":"https://api.github.com/users/ispirmustafa/followers","following_url":"https://api.github.com/users/ispirmustafa/following{/other_user}","gists_url":"https://api.github.com/users/ispirmustafa/gists{/gist_id}","starred_url":"https://api.github.com/users/ispirmustafa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ispirmustafa/subscriptions","organizations_url":"https://api.github.com/users/ispirmustafa/orgs","repos_url":"https://api.github.com/users/ispirmustafa/repos","events_url":"https://api.github.com/users/ispirmustafa/events{/privacy}","received_events_url":"https://api.github.com/users/ispirmustafa/received_events","type":"User","site_admin":false},"assignees":[{"login":"ispirmustafa","id":19293677,"node_id":"MDQ6VXNlcjE5MjkzNjc3","avatar_url":"https://avatars1.githubusercontent.com/u/19293677?v=4","gravatar_id":"","url":"https://api.github.com/users/ispirmustafa","html_url":"https://github.com/ispirmustafa","followers_url":"https://api.github.com/users/ispirmustafa/followers","following_url":"https://api.github.com/users/ispirmustafa/following{/other_user}","gists_url":"https://api.github.com/users/ispirmustafa/gists{/gist_id}","starred_url":"https://api.github.com/users/ispirmustafa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ispirmustafa/subscriptions","organizations_url":"https://api.github.com/users/ispirmustafa/orgs","repos_url":"https://api.github.com/users/ispirmustafa/repos","events_url":"https://api.github.com/users/ispirmustafa/events{/privacy}","received_events_url":"https://api.github.com/users/ispirmustafa/received_events","type":"User","site_admin":false}],"milestone":null,"comments":17,"created_at":"2018-05-03T16:26:11Z","updated_at":"2018-06-28T20:18:39Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: yes\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: N/A\r\n- **TensorFlow installed from (source or binary)**: binary\r\n- **TensorFlow version (use command below)**: v1.8.0-0-g93bc2e2072 1.8.0\r\n- **Python version**: 3.6\r\n- **Bazel version (if compiling from source)**: N/A\r\n- **GCC/Compiler version (if compiling from source)**: N/A\r\n- **CUDA/cuDNN version**: N/A\r\n- **GPU model and memory**: N/A\r\n- **Exact command to reproduce**: see below\r\n\r\n### Describe the problem\r\n\r\nWhen the input function is based on the one-shot dataset iterator, the training phase always starts from the beginning of the iterator. That is, the iterator state gets reset in between the train/eval phases. Therefore, if the dataset is big enough, then the training would only see a subset of the data, which can be processed in `eval_spec.throttle_secs`. \r\n\r\nI think the issue is caused by the fact that the graph is persisted before transitioning to the next phase, and restored upon reentering training. However, I find the behaviour a bit counterintuitive, so if it is not a bug, it should be mentioned in the `train_and_evaluate` docs.\r\n\r\n### Source code / logs\r\n\r\nHere is a small example demonstrating the issue:\r\n\r\n```python\r\nimport tensorflow as tf\r\n\r\n\r\ndef input_fn(data):\r\n    dataset = tf.data.Dataset.from_tensor_slices(data)\r\n    dataset = dataset.batch(batch_size=1)\r\n    x = dataset.make_one_shot_iterator().get_next()\r\n    return {\"x\": tf.Print(x, [x])}, x\r\n\r\n\r\nif __name__ == \"__main__\":\r\n    model = tf.estimator.LinearRegressor(feature_columns=[\r\n        tf.feature_column.numeric_column(\"x\")\r\n    ])\r\n\r\n    train_spec = tf.estimator.TrainSpec(\r\n        input_fn=lambda: input_fn(list(range(2**20))),\r\n        max_steps=2)\r\n    eval_spec = tf.estimator.EvalSpec(\r\n        input_fn=lambda: input_fn([42]),\r\n        steps=1,\r\n        start_delay_secs=1,\r\n        throttle_secs=1)\r\n\r\n    tf.logging.set_verbosity(\"INFO\")\r\n    tf.train.create_global_step()\r\n    tf.estimator.train_and_evaluate(model, train_spec, eval_spec)\r\n```\r\n\r\nThe code produces the following log output (I've omitted irrelevant lines):\r\n\r\n```\r\nINFO:tensorflow:Start train and evaluate loop. The evaluate will happen after 1 secs (eval_spec.throttle_secs) or training is finished.\r\nINFO:tensorflow:Calling model_fn.\r\nINFO:tensorflow:Done calling model_fn.\r\nINFO:tensorflow:Create CheckpointSaverHook.\r\nINFO:tensorflow:Graph was finalized.\r\nINFO:tensorflow:Running local_init_op.\r\nINFO:tensorflow:Done running local_init_op.\r\n[0]\r\nINFO:tensorflow:Saving checkpoints for 1 into /var/folders/wr/s7brqkzj74v4pmdwkwn19321l34xg2/T/tmpgtq1ap9_/model.ckpt.\r\nINFO:tensorflow:loss = 0.0, step = 1\r\nINFO:tensorflow:Loss for final step: 0.0.\r\n...\r\nINFO:tensorflow:Starting evaluation at 2018-05-03-16:21:51\r\n...\r\nINFO:tensorflow:Finished evaluation at 2018-05-03-16:21:51\r\n...\r\nINFO:tensorflow:Restoring parameters from /var/folders/wr/s7brqkzj74v4pmdwkwn19321l34xg2/T/tmpgtq1ap9_/model.ckpt-1\r\nINFO:tensorflow:Running local_init_op.\r\nINFO:tensorflow:Done running local_init_op.\r\n[0]\r\nINFO:tensorflow:Saving checkpoints for 2 into /var/folders/wr/s7brqkzj74v4pmdwkwn19321l34xg2/T/tmpgtq1ap9_/model.ckpt.\r\nINFO:tensorflow:loss = 0.0, step = 2\r\nINFO:tensorflow:Loss for final step: 0.0.\r\n...\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/401160323","html_url":"https://github.com/tensorflow/tensorflow/issues/19062#issuecomment-401160323","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19062","id":401160323,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTE2MDMyMw==","user":{"login":"superbobry","id":185856,"node_id":"MDQ6VXNlcjE4NTg1Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/185856?v=4","gravatar_id":"","url":"https://api.github.com/users/superbobry","html_url":"https://github.com/superbobry","followers_url":"https://api.github.com/users/superbobry/followers","following_url":"https://api.github.com/users/superbobry/following{/other_user}","gists_url":"https://api.github.com/users/superbobry/gists{/gist_id}","starred_url":"https://api.github.com/users/superbobry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/superbobry/subscriptions","organizations_url":"https://api.github.com/users/superbobry/orgs","repos_url":"https://api.github.com/users/superbobry/repos","events_url":"https://api.github.com/users/superbobry/events{/privacy}","received_events_url":"https://api.github.com/users/superbobry/received_events","type":"User","site_admin":false},"created_at":"2018-06-28T20:18:39Z","updated_at":"2018-06-28T20:18:39Z","author_association":"CONTRIBUTOR","body":"I'm sorry, I think I miss context to understand what needs to be hijacked. "}}
{"actor":{"display_login":"superbobry"}}