{"action":"created","issue":{"url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/20","repository_url":"https://api.github.com/repos/elliottd/GroundedTranslation","labels_url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/20/labels{/name}","comments_url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/20/comments","events_url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/20/events","html_url":"https://github.com/elliottd/GroundedTranslation/issues/20","id":149976451,"number":20,"title":"Generate.py doesn't work for test set","user":{"login":"evanmiltenburg","id":1513355,"avatar_url":"https://avatars.githubusercontent.com/u/1513355?v=3","gravatar_id":"","url":"https://api.github.com/users/evanmiltenburg","html_url":"https://github.com/evanmiltenburg","followers_url":"https://api.github.com/users/evanmiltenburg/followers","following_url":"https://api.github.com/users/evanmiltenburg/following{/other_user}","gists_url":"https://api.github.com/users/evanmiltenburg/gists{/gist_id}","starred_url":"https://api.github.com/users/evanmiltenburg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evanmiltenburg/subscriptions","organizations_url":"https://api.github.com/users/evanmiltenburg/orgs","repos_url":"https://api.github.com/users/evanmiltenburg/repos","events_url":"https://api.github.com/users/evanmiltenburg/events{/privacy}","received_events_url":"https://api.github.com/users/evanmiltenburg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-04-21T06:07:59Z","updated_at":"2016-04-21T06:15:59Z","closed_at":null,"body":"This kept me up yesterday: I was getting 507 sentences for the test set, rather than 1000. Looking at `generate.py` I found the problem was this part:\r\n\r\n```python\r\n                seen += text.shape[0]\r\n                if seen == self.data_gen.split_sizes['val']:\r\n                    # Hacky way to break out of the generator\r\n                    break\r\n```\r\nI fixed it by changing line 107 from `for data in generator:` to this:\r\n\r\n```python\r\n# Use enumerate() to count the number of data points, starting at 1.\r\nfor seen, data in enumerate(generator, start=1):\r\n```\r\n\r\nand lines 242-245 (the 'hacky way') to:\r\n```python\r\n                if seen == len(self.dataset[prefix]):\r\n                    # Hacky way to break out of the generator\r\n                    break\r\n```\r\nThis solution should now work for any part of the dataset you want to test on :)"},"comment":{"url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/comments/212761996","html_url":"https://github.com/elliottd/GroundedTranslation/issues/20#issuecomment-212761996","issue_url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/20","id":212761996,"user":{"login":"evanmiltenburg","id":1513355,"avatar_url":"https://avatars.githubusercontent.com/u/1513355?v=3","gravatar_id":"","url":"https://api.github.com/users/evanmiltenburg","html_url":"https://github.com/evanmiltenburg","followers_url":"https://api.github.com/users/evanmiltenburg/followers","following_url":"https://api.github.com/users/evanmiltenburg/following{/other_user}","gists_url":"https://api.github.com/users/evanmiltenburg/gists{/gist_id}","starred_url":"https://api.github.com/users/evanmiltenburg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evanmiltenburg/subscriptions","organizations_url":"https://api.github.com/users/evanmiltenburg/orgs","repos_url":"https://api.github.com/users/evanmiltenburg/repos","events_url":"https://api.github.com/users/evanmiltenburg/events{/privacy}","received_events_url":"https://api.github.com/users/evanmiltenburg/received_events","type":"User","site_admin":false},"created_at":"2016-04-21T06:15:59Z","updated_at":"2016-04-21T06:15:59Z","body":"Some other additions on my branch that I've found really useful are:\r\n\r\n* making an additional variable for the completed sentence, rather than use this multiple times:\r\n```python\r\n' '.join([x for x\r\n                                          in itertools.takewhile(\r\n                                              lambda n: n != \"<E>\",\r\n                                              complete_sentences[i])])\r\n```\r\n* Writing out JSON files with the image ID and the generated sentence. This required changing data_generator so that it also yields the ID:\r\n```python\r\n               yield {'text':arrays[0], 'img': arrays[1],\r\n                      'output': targets, 'ident': ident}\r\n```"}}