{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/13101","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/13101/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/13101/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/13101/events","html_url":"https://github.com/tensorflow/tensorflow/issues/13101","id":258314920,"node_id":"MDU6SXNzdWUyNTgzMTQ5MjA=","number":13101,"title":"from_generator feedback and questions","user":{"login":"tillahoffmann","id":966348,"node_id":"MDQ6VXNlcjk2NjM0OA==","avatar_url":"https://avatars2.githubusercontent.com/u/966348?v=4","gravatar_id":"","url":"https://api.github.com/users/tillahoffmann","html_url":"https://github.com/tillahoffmann","followers_url":"https://api.github.com/users/tillahoffmann/followers","following_url":"https://api.github.com/users/tillahoffmann/following{/other_user}","gists_url":"https://api.github.com/users/tillahoffmann/gists{/gist_id}","starred_url":"https://api.github.com/users/tillahoffmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tillahoffmann/subscriptions","organizations_url":"https://api.github.com/users/tillahoffmann/orgs","repos_url":"https://api.github.com/users/tillahoffmann/repos","events_url":"https://api.github.com/users/tillahoffmann/events{/privacy}","received_events_url":"https://api.github.com/users/tillahoffmann/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"mrry","id":192142,"node_id":"MDQ6VXNlcjE5MjE0Mg==","avatar_url":"https://avatars1.githubusercontent.com/u/192142?v=4","gravatar_id":"","url":"https://api.github.com/users/mrry","html_url":"https://github.com/mrry","followers_url":"https://api.github.com/users/mrry/followers","following_url":"https://api.github.com/users/mrry/following{/other_user}","gists_url":"https://api.github.com/users/mrry/gists{/gist_id}","starred_url":"https://api.github.com/users/mrry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrry/subscriptions","organizations_url":"https://api.github.com/users/mrry/orgs","repos_url":"https://api.github.com/users/mrry/repos","events_url":"https://api.github.com/users/mrry/events{/privacy}","received_events_url":"https://api.github.com/users/mrry/received_events","type":"User","site_admin":false},"assignees":[{"login":"mrry","id":192142,"node_id":"MDQ6VXNlcjE5MjE0Mg==","avatar_url":"https://avatars1.githubusercontent.com/u/192142?v=4","gravatar_id":"","url":"https://api.github.com/users/mrry","html_url":"https://github.com/mrry","followers_url":"https://api.github.com/users/mrry/followers","following_url":"https://api.github.com/users/mrry/following{/other_user}","gists_url":"https://api.github.com/users/mrry/gists{/gist_id}","starred_url":"https://api.github.com/users/mrry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrry/subscriptions","organizations_url":"https://api.github.com/users/mrry/orgs","repos_url":"https://api.github.com/users/mrry/repos","events_url":"https://api.github.com/users/mrry/events{/privacy}","received_events_url":"https://api.github.com/users/mrry/received_events","type":"User","site_admin":false}],"milestone":null,"comments":53,"created_at":"2017-09-17T16:25:38Z","updated_at":"2018-07-10T18:59:37Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"@mrry, thank you for implementing the `from_generator` method in `tf.data`. I just wanted to provide some feedback and ask a few more questions.\r\n\r\n# Interface\r\n\r\nIn addition to having `generator` be a callable that returns an iterator, would it be possible to support iterators that aren't wrapped in a callable? E.g. instead of \r\n\r\n```python\r\npool = multiprocessing.Pool()\r\ndataset = tf.contrib.data.Dataset.from_generator(\r\n    lambda: pool.imap(some_function, some_data), dtypes, shapes\r\n)\r\n```\r\n\r\nalso support\r\n\r\n```python\r\npool = multiprocessing.Pool()\r\ndataset = tf.contrib.data.Dataset.from_generator(\r\n    pool.imap(some_function, some_data), dtypes, shapes\r\n)\r\n```\r\n\r\n# Return types\r\n\r\n`from_generator` does not seem to support unpacking numpy arrays at the moment. I don't think it's essential but would be a nice-to-have. E.g. this fails\r\n\r\n```python\r\ndef generator():\r\n    while True:\r\n        yield np.zeros(2, np.float32)\r\n        \r\ndataset = tf.contrib.data.Dataset.from_generator(generator, (tf.float32, tf.float32))\r\nx, y = dataset.make_one_shot_iterator().get_next()\r\nsession = tf.Session()\r\nsession.run([x, y])\r\n```\r\n\r\nbut this runs smoothly\r\n\r\n```python\r\ndef generator():\r\n    while True:\r\n        yield tuple(np.zeros(2, np.float32))\r\n        \r\ndataset = tf.contrib.data.Dataset.from_generator(generator, (tf.float32, tf.float32))\r\nx, y = dataset.make_one_shot_iterator().get_next()\r\nsession = tf.Session()\r\nsession.run([x, y])\r\n```\r\n\r\n# Performance\r\n\r\nI've played around with a *very* naive example using the generator API (in the hope to eventually leverage ipyparallel or some other distributed computing framework). Unfortunately, I can't achieve the same performance that I would get using `feed_dicts`. The setup is as follows\r\n\r\n```python\r\nimport tensorflow as tf\r\nfrom tensorflow.contrib import data as tfdata\r\nimport numpy as np\r\nfrom time import time\r\n\r\nnum_batches = 1000\r\nbatch_size = 100\r\n\r\nclass Generator:\r\n    def __init__(self):\r\n        self.times = []\r\n    \r\n    def __iter__(self):\r\n        while True:\r\n            x = np.random.normal()\r\n            y = 3 + 5 * x\r\n            x, y = np.asarray([x, y], np.float32)\r\n            self.times.append(time())\r\n            yield x, y\r\n\r\ngenerator_state1 = Generator()\r\n\r\ndataset = tfdata.Dataset.from_generator(\r\n    lambda: generator_state1, \r\n    (tf.float32, tf.float32),\r\n    (tf.TensorShape([]), tf.TensorShape([]))\r\n)\r\nprefetched = dataset.prefetch(3 * batch_size)\r\nbatches = prefetched.batch(batch_size)\r\niterator = batches.make_one_shot_iterator()\r\n\r\nx, y = iterator.get_next()\r\n\r\nw = tf.Variable([0, 0], dtype=tf.float32)\r\nprediction = w[0] + w[1] * x\r\nloss = tf.losses.mean_squared_error(y, prediction)\r\noptimizer = tf.train.AdamOptimizer(0.1)\r\ntrain_op = optimizer.minimize(loss)\r\ninit_op = tf.global_variables_initializer()\r\n\r\nsession = tf.Session()\r\nsession.run(init_op)\r\n```\r\n\r\nRunning the optimisation gives me\r\n\r\n```python\r\nlosses = []\r\n\r\nstart = time()\r\nfor _ in range(num_batches):\r\n    _, _loss = session.run([train_op, loss])\r\n    losses.append(_loss)\r\ntime() - start  # about seven seconds\r\n```\r\n\r\nDoing the same using feed_dicts gives\r\n\r\n```python\r\nlosses = []\r\n\r\ngenerator_state2 = Generator()\r\niterator = iter(generator_state2)\r\n\r\nstart = time()\r\nfor _ in range(num_batches):\r\n    _x, _y = np.transpose([next(iterator) for _ in range(batch_size)])\r\n    _, _loss = session.run([train_op, loss], {x: _x, y: _y})\r\n    losses.append(_loss)\r\ntime() - start  # about one second\r\n```\r\n\r\nIt seems that the dataset created using the `from_generator` method isn't fetching from the generator fast enough:\r\n\r\n```python\r\nnp.mean(np.diff(generator_state1.times))  # 7.1533812683949508e-05\r\nnp.mean(np.diff(generator_state2.times))  # 1.0633696558370612e-05\r\n```\r\n\r\nThanks again for this, and looking forward to hearing your thoughts."},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/403931059","html_url":"https://github.com/tensorflow/tensorflow/issues/13101#issuecomment-403931059","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/13101","id":403931059,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMzkzMTA1OQ==","user":{"login":"tensorflowbutler","id":28546240,"node_id":"MDQ6VXNlcjI4NTQ2MjQw","avatar_url":"https://avatars2.githubusercontent.com/u/28546240?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflowbutler","html_url":"https://github.com/tensorflowbutler","followers_url":"https://api.github.com/users/tensorflowbutler/followers","following_url":"https://api.github.com/users/tensorflowbutler/following{/other_user}","gists_url":"https://api.github.com/users/tensorflowbutler/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflowbutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflowbutler/subscriptions","organizations_url":"https://api.github.com/users/tensorflowbutler/orgs","repos_url":"https://api.github.com/users/tensorflowbutler/repos","events_url":"https://api.github.com/users/tensorflowbutler/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflowbutler/received_events","type":"User","site_admin":false},"created_at":"2018-07-10T18:59:37Z","updated_at":"2018-07-10T18:59:37Z","author_association":"MEMBER","body":"Nagging Assignee @mrry: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly."}}
{"actor":{"display_login":"tensorflowbutler"}}