{"action":"created","issue":{"url":"https://api.github.com/repos/ModernMT/MMT/issues/184","repository_url":"https://api.github.com/repos/ModernMT/MMT","labels_url":"https://api.github.com/repos/ModernMT/MMT/issues/184/labels{/name}","comments_url":"https://api.github.com/repos/ModernMT/MMT/issues/184/comments","events_url":"https://api.github.com/repos/ModernMT/MMT/issues/184/events","html_url":"https://github.com/ModernMT/MMT/issues/184","id":187003260,"number":184,"title":"Big model use case","user":{"login":"vince62s","id":15141326,"avatar_url":"https://avatars.githubusercontent.com/u/15141326?v=3","gravatar_id":"","url":"https://api.github.com/users/vince62s","html_url":"https://github.com/vince62s","followers_url":"https://api.github.com/users/vince62s/followers","following_url":"https://api.github.com/users/vince62s/following{/other_user}","gists_url":"https://api.github.com/users/vince62s/gists{/gist_id}","starred_url":"https://api.github.com/users/vince62s/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vince62s/subscriptions","organizations_url":"https://api.github.com/users/vince62s/orgs","repos_url":"https://api.github.com/users/vince62s/repos","events_url":"https://api.github.com/users/vince62s/events{/privacy}","received_events_url":"https://api.github.com/users/vince62s/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"davidecaroselli","id":1674891,"avatar_url":"https://avatars.githubusercontent.com/u/1674891?v=3","gravatar_id":"","url":"https://api.github.com/users/davidecaroselli","html_url":"https://github.com/davidecaroselli","followers_url":"https://api.github.com/users/davidecaroselli/followers","following_url":"https://api.github.com/users/davidecaroselli/following{/other_user}","gists_url":"https://api.github.com/users/davidecaroselli/gists{/gist_id}","starred_url":"https://api.github.com/users/davidecaroselli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidecaroselli/subscriptions","organizations_url":"https://api.github.com/users/davidecaroselli/orgs","repos_url":"https://api.github.com/users/davidecaroselli/repos","events_url":"https://api.github.com/users/davidecaroselli/events{/privacy}","received_events_url":"https://api.github.com/users/davidecaroselli/received_events","type":"User","site_admin":false},"assignees":[{"login":"davidecaroselli","id":1674891,"avatar_url":"https://avatars.githubusercontent.com/u/1674891?v=3","gravatar_id":"","url":"https://api.github.com/users/davidecaroselli","html_url":"https://github.com/davidecaroselli","followers_url":"https://api.github.com/users/davidecaroselli/followers","following_url":"https://api.github.com/users/davidecaroselli/following{/other_user}","gists_url":"https://api.github.com/users/davidecaroselli/gists{/gist_id}","starred_url":"https://api.github.com/users/davidecaroselli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidecaroselli/subscriptions","organizations_url":"https://api.github.com/users/davidecaroselli/orgs","repos_url":"https://api.github.com/users/davidecaroselli/repos","events_url":"https://api.github.com/users/davidecaroselli/events{/privacy}","received_events_url":"https://api.github.com/users/davidecaroselli/received_events","type":"User","site_admin":false},{"login":"cattoni","id":1926528,"avatar_url":"https://avatars.githubusercontent.com/u/1926528?v=3","gravatar_id":"","url":"https://api.github.com/users/cattoni","html_url":"https://github.com/cattoni","followers_url":"https://api.github.com/users/cattoni/followers","following_url":"https://api.github.com/users/cattoni/following{/other_user}","gists_url":"https://api.github.com/users/cattoni/gists{/gist_id}","starred_url":"https://api.github.com/users/cattoni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cattoni/subscriptions","organizations_url":"https://api.github.com/users/cattoni/orgs","repos_url":"https://api.github.com/users/cattoni/repos","events_url":"https://api.github.com/users/cattoni/events{/privacy}","received_events_url":"https://api.github.com/users/cattoni/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2016-11-03T08:48:55Z","updated_at":"2016-11-03T16:36:49Z","closed_at":null,"body":"Hi,\r\nI successfully built an engine with 9 tmx sources totalling 23 Mio segments, and one text source (news shuffle) of 1.5 GB for monolingual extra data.\r\nthe engine folder totals 50 GB of which 15GB for the LM and 35 GB for the Sapt files.\r\n\r\nI kept increasing the memory up to 90 GB of ram.\r\n\r\nNever loaded the model successfully.\r\n\r\nI am rebuilding with less data right now.\r\n\r\nJust curious if you guys loaded such big size models and how long it takes."},"comment":{"url":"https://api.github.com/repos/ModernMT/MMT/issues/comments/258198366","html_url":"https://github.com/ModernMT/MMT/issues/184#issuecomment-258198366","issue_url":"https://api.github.com/repos/ModernMT/MMT/issues/184","id":258198366,"user":{"login":"davidecaroselli","id":1674891,"avatar_url":"https://avatars.githubusercontent.com/u/1674891?v=3","gravatar_id":"","url":"https://api.github.com/users/davidecaroselli","html_url":"https://github.com/davidecaroselli","followers_url":"https://api.github.com/users/davidecaroselli/followers","following_url":"https://api.github.com/users/davidecaroselli/following{/other_user}","gists_url":"https://api.github.com/users/davidecaroselli/gists{/gist_id}","starred_url":"https://api.github.com/users/davidecaroselli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidecaroselli/subscriptions","organizations_url":"https://api.github.com/users/davidecaroselli/orgs","repos_url":"https://api.github.com/users/davidecaroselli/repos","events_url":"https://api.github.com/users/davidecaroselli/events{/privacy}","received_events_url":"https://api.github.com/users/davidecaroselli/received_events","type":"User","site_admin":false},"created_at":"2016-11-03T16:36:49Z","updated_at":"2016-11-03T16:36:49Z","body":"@vince62s \r\n\r\nThanks for the feedback on memory requirements, we will update that info.\r\nFor the ratio between disk space / RAM space: with the new infrastructure of MMT there is no more 1-to-1 ratio.\r\nAt the moment the LM is the one that definitely takes much RAM (both because of the monolingual data, and the optimization needed for the AdaptiveLM). While the SAPT are much less greedy of RAM.\r\nPlease notice also that lots of data-structures are memory mapped, this means that the OS keep them in RAM if it has enough free space (instead of waste free space, it tends to use it as much as it can). So it can be possible that on large machines MMT will try to occupy RAM if it's free in order to speed up the computation.\r\n\r\nGenerally speaking, this release is not a low-RAM engine yet. But in the future we will work to make the system more efficient for sure!\r\n\r\nThank you."}}
{"actor":{"display_login":"davidecaroselli"}}