{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2514","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2514/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2514/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2514/events","html_url":"https://github.com/tensorflow/tensorflow/issues/2514","id":157033373,"number":2514,"title":"Feature Request: multi-epoch alternative to tf.QueueBase.close()","user":{"login":"markpwoodward","id":6820773,"avatar_url":"https://avatars.githubusercontent.com/u/6820773?v=3","gravatar_id":"","url":"https://api.github.com/users/markpwoodward","html_url":"https://github.com/markpwoodward","followers_url":"https://api.github.com/users/markpwoodward/followers","following_url":"https://api.github.com/users/markpwoodward/following{/other_user}","gists_url":"https://api.github.com/users/markpwoodward/gists{/gist_id}","starred_url":"https://api.github.com/users/markpwoodward/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markpwoodward/subscriptions","organizations_url":"https://api.github.com/users/markpwoodward/orgs","repos_url":"https://api.github.com/users/markpwoodward/repos","events_url":"https://api.github.com/users/markpwoodward/events{/privacy}","received_events_url":"https://api.github.com/users/markpwoodward/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"ebrevdo","id":1794715,"avatar_url":"https://avatars.githubusercontent.com/u/1794715?v=3","gravatar_id":"","url":"https://api.github.com/users/ebrevdo","html_url":"https://github.com/ebrevdo","followers_url":"https://api.github.com/users/ebrevdo/followers","following_url":"https://api.github.com/users/ebrevdo/following{/other_user}","gists_url":"https://api.github.com/users/ebrevdo/gists{/gist_id}","starred_url":"https://api.github.com/users/ebrevdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ebrevdo/subscriptions","organizations_url":"https://api.github.com/users/ebrevdo/orgs","repos_url":"https://api.github.com/users/ebrevdo/repos","events_url":"https://api.github.com/users/ebrevdo/events{/privacy}","received_events_url":"https://api.github.com/users/ebrevdo/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2016-05-26T17:12:27Z","updated_at":"2016-05-26T17:13:49Z","closed_at":null,"body":"Examples on the web demonstrate signaling the end of queue data by calling queue.close() in the data producer and then catching the tf.errors.OutOfRangeError exception in the data consumer.\r\n\r\nThis works fine for a single epoch, but I do multiple epochs, alternating between training data and testing data, and I can't reuse the queue after calling queue.close().\r\n\r\nThe two solutions that I have thought of using the existing code are:\r\n1) enqueue() some sentinel at the end of an epoch in the data producer and then tf.Assert() against the sentinel and catch the tf.errors.InvalidArgumentError in the data consumer.\r\n2) know the number of enqueue's for the epoch and only dequeue that number.\r\nBoth seem a little hacky.\r\n\r\nMulti-epoch use of queues might be simplified by adding one of the following:\r\n1) A queue.reset(), that throws one tf.errors.OutOfRangeError on dequeue() or some other exception.\r\n2) A queue.close(reset=True), that only throws one tf.errors.OutOfRangeError on dequeue() or some other exception.\r\n\r\nexample usage of 1):\r\n```\r\nq = tf.FIFOQueue(...)\r\nplaceholder = ...\r\nenqueue_op = q.enqueue(placeholder)\r\n....\r\n\r\ndef producer(data_dir, sess, q, enqueue_op, placeholder):\r\n  for ...:\r\n    sess.run(enqueue_op, {placeholder:...})\r\n  sess.run(q.reset())\r\n\r\ndef do_epoch(data_dir, learn):\r\n  threading.Thread(target=producer, args=(data_dir, sess, q, enqueue_op, placeholder)).start()\r\n  while True:\r\n    try:\r\n      sess.run(...)\r\n    exception tf.errors.OutOfRangeError:\r\n      break\r\n\r\nfor epoch in range(NUM_EPOCHS):\r\n  ... = do_epoch(TRAIN_DIR, learn=True)\r\n  ... = do_epoch(TEST_DIR, learn=False)\r\n```"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/221934925","html_url":"https://github.com/tensorflow/tensorflow/issues/2514#issuecomment-221934925","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/2514","id":221934925,"user":{"login":"vrv","id":463737,"avatar_url":"https://avatars.githubusercontent.com/u/463737?v=3","gravatar_id":"","url":"https://api.github.com/users/vrv","html_url":"https://github.com/vrv","followers_url":"https://api.github.com/users/vrv/followers","following_url":"https://api.github.com/users/vrv/following{/other_user}","gists_url":"https://api.github.com/users/vrv/gists{/gist_id}","starred_url":"https://api.github.com/users/vrv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vrv/subscriptions","organizations_url":"https://api.github.com/users/vrv/orgs","repos_url":"https://api.github.com/users/vrv/repos","events_url":"https://api.github.com/users/vrv/events{/privacy}","received_events_url":"https://api.github.com/users/vrv/received_events","type":"User","site_admin":false},"created_at":"2016-05-26T17:13:49Z","updated_at":"2016-05-26T17:13:49Z","body":"cc @ebrevdo @mrry @josh11b "}}
{"actor":{"display_login":"vrv"}}