{"action":"opened","issue":{"url":"https://api.github.com/repos/harvardnlp/seq2seq-attn/issues/72","repository_url":"https://api.github.com/repos/harvardnlp/seq2seq-attn","labels_url":"https://api.github.com/repos/harvardnlp/seq2seq-attn/issues/72/labels{/name}","comments_url":"https://api.github.com/repos/harvardnlp/seq2seq-attn/issues/72/comments","events_url":"https://api.github.com/repos/harvardnlp/seq2seq-attn/issues/72/events","html_url":"https://github.com/harvardnlp/seq2seq-attn/issues/72","id":192515592,"number":72,"title":"Memory requirements in preprocessing","user":{"login":"oraveczcsaba","id":7645311,"avatar_url":"https://avatars.githubusercontent.com/u/7645311?v=3","gravatar_id":"","url":"https://api.github.com/users/oraveczcsaba","html_url":"https://github.com/oraveczcsaba","followers_url":"https://api.github.com/users/oraveczcsaba/followers","following_url":"https://api.github.com/users/oraveczcsaba/following{/other_user}","gists_url":"https://api.github.com/users/oraveczcsaba/gists{/gist_id}","starred_url":"https://api.github.com/users/oraveczcsaba/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oraveczcsaba/subscriptions","organizations_url":"https://api.github.com/users/oraveczcsaba/orgs","repos_url":"https://api.github.com/users/oraveczcsaba/repos","events_url":"https://api.github.com/users/oraveczcsaba/events{/privacy}","received_events_url":"https://api.github.com/users/oraveczcsaba/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-11-30T09:28:59Z","updated_at":"2016-11-30T09:28:59Z","closed_at":null,"body":"The preprocess.py script initializes a couple of matrices for various data storage. For big training datasets (we are now trying to train with some 12M segments) this seems to need large amounts of memory, especially if we want to use guided alignment (I might not be right but I would roughly estimate it to hundreds of GB:\r\nalignments = np.zeros((num_sents,newseqlength,newseqlength), dtype=np.uint8)\r\nwith 12M segments, and a maximum length of about 80 tokens per segment). \r\n\r\nWould there be some quick and easy way of avoiding the MemoryError we get here and running such a training with some 64 GB of memory only?\r\n"}}
{"actor":{"display_login":"oraveczcsaba"}}