{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/7747","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/7747/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/7747/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/7747/events","html_url":"https://github.com/tensorflow/tensorflow/issues/7747","id":209246815,"number":7747,"title":"Why restore checkpoint will effect the tfrecord reader path? ","user":{"login":"ericyue","id":918889,"avatar_url":"https://avatars0.githubusercontent.com/u/918889?v=4","gravatar_id":"","url":"https://api.github.com/users/ericyue","html_url":"https://github.com/ericyue","followers_url":"https://api.github.com/users/ericyue/followers","following_url":"https://api.github.com/users/ericyue/following{/other_user}","gists_url":"https://api.github.com/users/ericyue/gists{/gist_id}","starred_url":"https://api.github.com/users/ericyue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericyue/subscriptions","organizations_url":"https://api.github.com/users/ericyue/orgs","repos_url":"https://api.github.com/users/ericyue/repos","events_url":"https://api.github.com/users/ericyue/events{/privacy}","received_events_url":"https://api.github.com/users/ericyue/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-02-21T19:23:27Z","updated_at":"2018-03-27T11:07:06Z","closed_at":"2017-02-22T05:38:03Z","author_association":"NONE","body":"I'm training my model with training dataset in standard format\r\n```\r\nfeatures = tf.parse_example(\r\n    batch_serialized_example,\r\n    features={\r\n        \"label\": tf.FixedLenFeature([], tf.float32),\r\n        \"ids\": tf.VarLenFeature(tf.int64),\r\n        \"values\": tf.VarLenFeature(tf.float32),\r\n    })\r\nbatch_labels = features[\"label\"]\r\nbatch_ids = features[\"ids\"]\r\nbatch_values = features[\"values\"]\r\n```\r\nthen I  save the checkpoint and want to do some predicting on my other dataset, **but add two more fields: uid and item_id (just for statistics)**\r\n\r\n```\r\npredict_features = tf.parse_example(\r\n    predict_batch_serialized_example,\r\n    features={\r\n        \"label\": tf.FixedLenFeature([], tf.float32),\r\n        \"ids\": tf.VarLenFeature(tf.int64),\r\n        \"values\": tf.VarLenFeature(tf.float32),\r\n        \"uid\": tf.FixedLenFeature([], tf.int64),\r\n        \"item_id\": tf.FixedLenFeature([], tf.int64)\r\n    })\r\npredict_batch_labels = predict_features[\"label\"]\r\npredict_batch_ids = predict_features[\"ids\"]\r\npredict_batch_values = predict_features[\"values\"]\r\npredict_batch_uids = predict_features[\"uid\"]\r\npredict_batch_item_ids = predict_features[\"item_id\"]\r\n\r\n```\r\n\r\nWhen I restore the checkpoint , the simply read-data  faild `InvalidArgumentError (see above for traceback): Name: <unknown>, Feature: item_id (data type: int64) is required but could not be found.\r\n`\r\nWhen I remove the restore function , it's seems right.\r\n\r\nWhy the checkpoint do effect the reader? it's seems that after restore the checkponit ,the reader no longer read from the new path ?\r\n\r\n### If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)\r\n\r\nhttps://gist.github.com/ericyue/e694a90338b9fadf9996025719005c9d\r\n\r\nmininal code (just read data)\r\n\r\n```\r\n      if not restore_session_from_checkpoint(sess, saver,cf):\r\n        logging.error(\"No checkpoint found, exit now\")\r\n        exit(1)\r\n      try:\r\n          while not coord.should_stop():\r\n            uids, item_ids = sess.run( [predict_batch_uids, predict_batch_item_ids])\r\n      except tf.errors.OutOfRangeError:\r\n          logging.info(\"Done predicting after reading all data\")\r\n      finally:\r\n          coord.request_stop()\r\n          logging.info(\"coord stopped\")\r\n```\r\n\r\n### Logs or other output that would be helpful\r\n(If logs are large, please upload as attachment or provide link).\r\n```\r\n  1 Caused by op u'ParseExample_2/ParseExample', defined at:\r\n  2   File \"deepcake-tf.py\", line 139, in <module>\r\n  3     \"item_id\": tf.FixedLenFeature([], tf.int64)\r\n  4   File \"/home/moon/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/parsing_ops.py\", line 445, in parse_example\r\n  5     dense_types, dense_defaults, dense_shapes, name)\r\n  6   File \"/home/moon/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/parsing_ops.py\", line 544, in _parse_example_raw\r\n  7     name=name)\r\n  8   File \"/home/moon/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_parsing_ops.py\", line 167, in _parse_example\r\n  9     dense_shapes=dense_shapes, name=name)\r\n 10   File \"/home/moon/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py\", line 763, in apply_op\r\n 11     op_def=op_def)\r\n 12   File \"/home/moon/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py\", line 2395, in create_op\r\n 13     original_op=self._default_original_op, op_def=op_def)\r\n 14   File \"/home/moon/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py\", line 1264, in __init__\r\n 15     self._traceback = _extract_stack()\r\n\r\nInvalidArgumentError (see above for traceback): Name: <unknown>, Feature: item_id (data type: int64) is required but could not be found.\r\n\t [[Node: ParseExample_2/ParseExample = ParseExample[Ndense=3, Nsparse=2, Tdense=[DT_INT64, DT_FLOAT, DT_INT64], dense_shapes=[[], [], []], sparse_types=[DT_INT64, DT_FLOAT], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](shuffle_batch_2, ParseExample_2/ParseExample/names, ParseExample_2/ParseExample/sparse_keys_0, ParseExample_2/ParseExample/sparse_keys_1, ParseExample_2/ParseExample/dense_keys_0, ParseExample_2/ParseExample/dense_keys_1, ParseExample_2/ParseExample/dense_keys_2, ParseExample_2/Const, ParseExample_2/Const_1, ParseExample_2/Const_2)]]\r\n```"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/376485693","html_url":"https://github.com/tensorflow/tensorflow/issues/7747#issuecomment-376485693","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/7747","id":376485693,"user":{"login":"b01902041","id":14891333,"avatar_url":"https://avatars3.githubusercontent.com/u/14891333?v=4","gravatar_id":"","url":"https://api.github.com/users/b01902041","html_url":"https://github.com/b01902041","followers_url":"https://api.github.com/users/b01902041/followers","following_url":"https://api.github.com/users/b01902041/following{/other_user}","gists_url":"https://api.github.com/users/b01902041/gists{/gist_id}","starred_url":"https://api.github.com/users/b01902041/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b01902041/subscriptions","organizations_url":"https://api.github.com/users/b01902041/orgs","repos_url":"https://api.github.com/users/b01902041/repos","events_url":"https://api.github.com/users/b01902041/events{/privacy}","received_events_url":"https://api.github.com/users/b01902041/received_events","type":"User","site_admin":false},"created_at":"2018-03-27T11:07:06Z","updated_at":"2018-03-27T11:07:06Z","author_association":"NONE","body":"Hi, how did you fix it?"}}
{"actor":{"display_login":"b01902041"}}