{"action":"created","issue":{"url":"https://api.github.com/repos/marian-nmt/marian-dev/issues/148","repository_url":"https://api.github.com/repos/marian-nmt/marian-dev","labels_url":"https://api.github.com/repos/marian-nmt/marian-dev/issues/148/labels{/name}","comments_url":"https://api.github.com/repos/marian-nmt/marian-dev/issues/148/comments","events_url":"https://api.github.com/repos/marian-nmt/marian-dev/issues/148/events","html_url":"https://github.com/marian-nmt/marian-dev/issues/148","id":278764110,"node_id":"MDU6SXNzdWUyNzg3NjQxMTA=","number":148,"title":"Don't load entire corpus into memory on start up (enhancement request)","user":{"login":"bhaddow","id":992795,"node_id":"MDQ6VXNlcjk5Mjc5NQ==","avatar_url":"https://avatars1.githubusercontent.com/u/992795?v=4","gravatar_id":"","url":"https://api.github.com/users/bhaddow","html_url":"https://github.com/bhaddow","followers_url":"https://api.github.com/users/bhaddow/followers","following_url":"https://api.github.com/users/bhaddow/following{/other_user}","gists_url":"https://api.github.com/users/bhaddow/gists{/gist_id}","starred_url":"https://api.github.com/users/bhaddow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bhaddow/subscriptions","organizations_url":"https://api.github.com/users/bhaddow/orgs","repos_url":"https://api.github.com/users/bhaddow/repos","events_url":"https://api.github.com/users/bhaddow/events{/privacy}","received_events_url":"https://api.github.com/users/bhaddow/received_events","type":"User","site_admin":false},"labels":[{"id":370545880,"node_id":"MDU6TGFiZWwzNzA1NDU4ODA=","url":"https://api.github.com/repos/marian-nmt/marian-dev/labels/enhancement","name":"enhancement","color":"1d76db","default":true},{"id":370545881,"node_id":"MDU6TGFiZWwzNzA1NDU4ODE=","url":"https://api.github.com/repos/marian-nmt/marian-dev/labels/help%20wanted","name":"help wanted","color":"128A0C","default":true}],"state":"closed","locked":false,"assignee":{"login":"emjotde","id":1083125,"node_id":"MDQ6VXNlcjEwODMxMjU=","avatar_url":"https://avatars3.githubusercontent.com/u/1083125?v=4","gravatar_id":"","url":"https://api.github.com/users/emjotde","html_url":"https://github.com/emjotde","followers_url":"https://api.github.com/users/emjotde/followers","following_url":"https://api.github.com/users/emjotde/following{/other_user}","gists_url":"https://api.github.com/users/emjotde/gists{/gist_id}","starred_url":"https://api.github.com/users/emjotde/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emjotde/subscriptions","organizations_url":"https://api.github.com/users/emjotde/orgs","repos_url":"https://api.github.com/users/emjotde/repos","events_url":"https://api.github.com/users/emjotde/events{/privacy}","received_events_url":"https://api.github.com/users/emjotde/received_events","type":"User","site_admin":false},"assignees":[{"login":"emjotde","id":1083125,"node_id":"MDQ6VXNlcjEwODMxMjU=","avatar_url":"https://avatars3.githubusercontent.com/u/1083125?v=4","gravatar_id":"","url":"https://api.github.com/users/emjotde","html_url":"https://github.com/emjotde","followers_url":"https://api.github.com/users/emjotde/followers","following_url":"https://api.github.com/users/emjotde/following{/other_user}","gists_url":"https://api.github.com/users/emjotde/gists{/gist_id}","starred_url":"https://api.github.com/users/emjotde/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emjotde/subscriptions","organizations_url":"https://api.github.com/users/emjotde/orgs","repos_url":"https://api.github.com/users/emjotde/repos","events_url":"https://api.github.com/users/emjotde/events{/privacy}","received_events_url":"https://api.github.com/users/emjotde/received_events","type":"User","site_admin":false}],"milestone":null,"comments":22,"created_at":"2017-12-03T11:06:05Z","updated_at":"2018-12-16T17:45:33Z","closed_at":"2018-01-24T20:42:15Z","author_association":"CONTRIBUTOR","body":"GPU servers often don't have much RAM, so loading the full corpus into memory for shuffling limits the size of the corpus that can be used for training. Particularly if there are other jobs on the machine. Or if you want to start several marian jobs at a time.\r\n\r\nI realise that this got discussed in other thread, but I think it was OT, and the thread was closed as the main issue was fixed."},"comment":{"url":"https://api.github.com/repos/marian-nmt/marian-dev/issues/comments/447662076","html_url":"https://github.com/marian-nmt/marian-dev/issues/148#issuecomment-447662076","issue_url":"https://api.github.com/repos/marian-nmt/marian-dev/issues/148","id":447662076,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NzY2MjA3Ng==","user":{"login":"emjotde","id":1083125,"node_id":"MDQ6VXNlcjEwODMxMjU=","avatar_url":"https://avatars3.githubusercontent.com/u/1083125?v=4","gravatar_id":"","url":"https://api.github.com/users/emjotde","html_url":"https://github.com/emjotde","followers_url":"https://api.github.com/users/emjotde/followers","following_url":"https://api.github.com/users/emjotde/following{/other_user}","gists_url":"https://api.github.com/users/emjotde/gists{/gist_id}","starred_url":"https://api.github.com/users/emjotde/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emjotde/subscriptions","organizations_url":"https://api.github.com/users/emjotde/orgs","repos_url":"https://api.github.com/users/emjotde/repos","events_url":"https://api.github.com/users/emjotde/events{/privacy}","received_events_url":"https://api.github.com/users/emjotde/received_events","type":"User","site_admin":false},"created_at":"2018-12-16T17:45:33Z","updated_at":"2018-12-16T17:45:33Z","author_association":"MEMBER","body":"It’s a trade off I suppose. Token ids would take ages in the construction phase. Currently, you should only notice the parsing time into tokens during the very first pre-read as it cannot do any computation without having the data. Later there is no slow-down as we pre-read in parallel to computation and prereading is usually done much faster, so there is no idle time. If there is no idle time there is no potential for speed-up.\n\nFrom: Jim Geovedi\nSent: Sunday, December 16, 2018 4:53 AM\nTo: marian-nmt/marian-dev\nCc: Marcin Junczys-Dowmunt; Mention\nSubject: Re: [marian-nmt/marian-dev] Don't load entire corpus into memory onstart up (enhancement request) (#148)\n\n@emjotde i'm testing --sqlite option and noticed that plain text is stored. would storing token ids improve speed instead of plain text?\ni'm training 520m corpus here, can confirm the memory usage is low.\n—\nYou are receiving this because you were mentioned.\nReply to this email directly, view it on GitHub, or mute the thread.\n\n"}}
{"actor":{"display_login":"emjotde"}}