{"action":"created","issue":{"url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/17","repository_url":"https://api.github.com/repos/elliottd/GroundedTranslation","labels_url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/17/comments","events_url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/17/events","html_url":"https://github.com/elliottd/GroundedTranslation/issues/17","id":140969947,"number":17,"title":"MemoryError and rewriting data_generator","user":{"login":"evanmiltenburg","id":1513355,"avatar_url":"https://avatars.githubusercontent.com/u/1513355?v=3","gravatar_id":"","url":"https://api.github.com/users/evanmiltenburg","html_url":"https://github.com/evanmiltenburg","followers_url":"https://api.github.com/users/evanmiltenburg/followers","following_url":"https://api.github.com/users/evanmiltenburg/following{/other_user}","gists_url":"https://api.github.com/users/evanmiltenburg/gists{/gist_id}","starred_url":"https://api.github.com/users/evanmiltenburg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evanmiltenburg/subscriptions","organizations_url":"https://api.github.com/users/evanmiltenburg/orgs","repos_url":"https://api.github.com/users/evanmiltenburg/repos","events_url":"https://api.github.com/users/evanmiltenburg/events{/privacy}","received_events_url":"https://api.github.com/users/evanmiltenburg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-03-15T13:21:24Z","updated_at":"2016-03-16T10:35:00Z","closed_at":null,"body":"Since I don't want to bombard your inbox with emails, I'll make an issue here for the MemoryError I got with the Flickr30k data. Loading everything at once into memory seems to be too cumbersome for my machine. So maybe a generator-based approach is better. Keras now provides a `fit_generator` function that seems perfect to keep memory requirements as low as possible.\r\n\r\nI just wrote a simple generator that automatically randomizes the training and val data, see 6dd0b27215594d6602036588fbfd4d6379bb59e1. If I can get everything to run, this should make the module scale really well."},"comment":{"url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/comments/197253778","html_url":"https://github.com/elliottd/GroundedTranslation/issues/17#issuecomment-197253778","issue_url":"https://api.github.com/repos/elliottd/GroundedTranslation/issues/17","id":197253778,"user":{"login":"evanmiltenburg","id":1513355,"avatar_url":"https://avatars.githubusercontent.com/u/1513355?v=3","gravatar_id":"","url":"https://api.github.com/users/evanmiltenburg","html_url":"https://github.com/evanmiltenburg","followers_url":"https://api.github.com/users/evanmiltenburg/followers","following_url":"https://api.github.com/users/evanmiltenburg/following{/other_user}","gists_url":"https://api.github.com/users/evanmiltenburg/gists{/gist_id}","starred_url":"https://api.github.com/users/evanmiltenburg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evanmiltenburg/subscriptions","organizations_url":"https://api.github.com/users/evanmiltenburg/orgs","repos_url":"https://api.github.com/users/evanmiltenburg/repos","events_url":"https://api.github.com/users/evanmiltenburg/events{/privacy}","received_events_url":"https://api.github.com/users/evanmiltenburg/received_events","type":"User","site_admin":false},"created_at":"2016-03-16T10:35:00Z","updated_at":"2016-03-16T10:35:00Z","body":"> I suspect it happens more when you're generating because you've already filled up most of your ram with perplexity calculations and python isn't garbage collecting hyperefficiently, so you start swapping.\r\n\r\nAh, so adding a couple of `del` statements (i.e. actively stepping in to manage the memory) might already help a lot in terms of efficiency.\r\n\r\n> Does Keras now support passing generators for the validation data to keras.fit()?\r\n\r\nYes, or at least there's a new function called `fit_generator`. It has arguments for both a training data generator as well as a validation data generator. [Link to models.py, which has very informative docstrings](https://github.com/fchollet/keras/blob/master/keras/models.py).\r\n\r\n> For your current experiments: can you cut your validation data down to a very small number (50-100 sentences) and train on --unk 3 or so? I.e. is the bottleneck for the vocabulary just due to the val array sizes?\r\n\r\nGood idea. Will try that later today."}}