{"action":"created","issue":{"url":"https://api.github.com/repos/maxhodak/keras-molecules/issues/35","repository_url":"https://api.github.com/repos/maxhodak/keras-molecules","labels_url":"https://api.github.com/repos/maxhodak/keras-molecules/issues/35/labels{/name}","comments_url":"https://api.github.com/repos/maxhodak/keras-molecules/issues/35/comments","events_url":"https://api.github.com/repos/maxhodak/keras-molecules/issues/35/events","html_url":"https://github.com/maxhodak/keras-molecules/issues/35","id":189250457,"number":35,"title":"Preprocess uses too much RAM","user":{"login":"dakoner","id":8090176,"avatar_url":"https://avatars.githubusercontent.com/u/8090176?v=3","gravatar_id":"","url":"https://api.github.com/users/dakoner","html_url":"https://github.com/dakoner","followers_url":"https://api.github.com/users/dakoner/followers","following_url":"https://api.github.com/users/dakoner/following{/other_user}","gists_url":"https://api.github.com/users/dakoner/gists{/gist_id}","starred_url":"https://api.github.com/users/dakoner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dakoner/subscriptions","organizations_url":"https://api.github.com/users/dakoner/orgs","repos_url":"https://api.github.com/users/dakoner/repos","events_url":"https://api.github.com/users/dakoner/events{/privacy}","received_events_url":"https://api.github.com/users/dakoner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":18,"created_at":"2016-11-14T22:56:56Z","updated_at":"2016-11-23T16:58:50Z","closed_at":null,"body":"preprocess.py loads the entire pre-preprocessed data into RAM, does transforms that require more RAM.  I'm trying to preprocess GDB-17 w/ 50M SMILES strings and it's just about filling up my 64GB RAM machine.  We should be able to go directly from SMILES input to preprocessed input files with far fewer resources although it would take work to make all the steps incremental."},"comment":{"url":"https://api.github.com/repos/maxhodak/keras-molecules/issues/comments/262572204","html_url":"https://github.com/maxhodak/keras-molecules/issues/35#issuecomment-262572204","issue_url":"https://api.github.com/repos/maxhodak/keras-molecules/issues/35","id":262572204,"user":{"login":"dakoner","id":8090176,"avatar_url":"https://avatars.githubusercontent.com/u/8090176?v=3","gravatar_id":"","url":"https://api.github.com/users/dakoner","html_url":"https://github.com/dakoner","followers_url":"https://api.github.com/users/dakoner/followers","following_url":"https://api.github.com/users/dakoner/following{/other_user}","gists_url":"https://api.github.com/users/dakoner/gists{/gist_id}","starred_url":"https://api.github.com/users/dakoner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dakoner/subscriptions","organizations_url":"https://api.github.com/users/dakoner/orgs","repos_url":"https://api.github.com/users/dakoner/repos","events_url":"https://api.github.com/users/dakoner/events{/privacy}","received_events_url":"https://api.github.com/users/dakoner/received_events","type":"User","site_admin":false},"created_at":"2016-11-23T16:58:50Z","updated_at":"2016-11-23T16:58:50Z","body":"In this case, I'm training on a file with 1.3M compounds.  I believe your\ncomment only applies if I'm providing a small input file and setting\n--epoch_size to a value that is close to the input file size... right?\nAnyway, seems like I should be passing --test_split as a parameter, and the\ncode should do the work of figuring out everything else.\n\nOn Wed, Nov 23, 2016 at 8:56 AM, Yakov Pechersky <notifications@github.com>\nwrote:\n\n> I should add that if you are training on 35K, and you assume the default\n> test_split=0.20, then your \"true\" effective training set size is 28K.\n> That's the epoch size you'll want to use, which will also give you a\n> test-epoch of 7K.\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/maxhodak/keras-molecules/issues/35#issuecomment-262571441>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AHtyQMOKsAV9yR80fGvHgyP6oxR_eu1Cks5rBHAhgaJpZM4Kx6jM>\n> .\n>\n"}}
{"actor":{"display_login":"dakoner"}}