{"action":"created","issue":{"url":"https://api.github.com/repos/kpu/kenlm/issues/177","repository_url":"https://api.github.com/repos/kpu/kenlm","labels_url":"https://api.github.com/repos/kpu/kenlm/issues/177/labels{/name}","comments_url":"https://api.github.com/repos/kpu/kenlm/issues/177/comments","events_url":"https://api.github.com/repos/kpu/kenlm/issues/177/events","html_url":"https://github.com/kpu/kenlm/issues/177","id":355057015,"node_id":"MDU6SXNzdWUzNTUwNTcwMTU=","number":177,"title":"Building intermediate models with a predefined vocabulary leads to \"poison\" error","user":{"login":"geniki","id":13801078,"node_id":"MDQ6VXNlcjEzODAxMDc4","avatar_url":"https://avatars1.githubusercontent.com/u/13801078?v=4","gravatar_id":"","url":"https://api.github.com/users/geniki","html_url":"https://github.com/geniki","followers_url":"https://api.github.com/users/geniki/followers","following_url":"https://api.github.com/users/geniki/following{/other_user}","gists_url":"https://api.github.com/users/geniki/gists{/gist_id}","starred_url":"https://api.github.com/users/geniki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/geniki/subscriptions","organizations_url":"https://api.github.com/users/geniki/orgs","repos_url":"https://api.github.com/users/geniki/repos","events_url":"https://api.github.com/users/geniki/events{/privacy}","received_events_url":"https://api.github.com/users/geniki/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-08-29T08:42:41Z","updated_at":"2019-07-26T02:31:55Z","closed_at":null,"author_association":"NONE","body":"I’d like to build intermediate models with a predefined vocabulary and then interpolate them. However, I’m getting the following error at the last step of building the intermediate models.\r\n```\r\n\r\n...\r\n=== 4/4 Calculating and writing order-interpolated probabilities ===\r\nChain sizes: 1:8904 2:50640\r\n----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100\r\n----------------------------------------------------------------------------------------------------\r\nLast input should have been poison.\r\n```\r\n\r\nThe models actually build fine if I do not use a vocabulary file and do not use the --prune option. A vocabulary file that contains all tokens in the corpus (so no effect) is also OK. Arpa output with any pruning/vocabulary works as expected.\r\n\r\nI tried lower LM order (down to 2), decreasing the corpus size (to ~10k words) and a different dataset but none of these make a difference. The machine has ~500GB disk space (path set using -T) and ~50GB RAM.\r\n\r\nExample command:\r\n`kenlm lmplz -o 2 --text corpus.txt --intermediate my_new_lm -T /path_to_500GB --limit_vocab_file vocab.txt\r\n`\r\nAny thoughts on how to address this?"},"comment":{"url":"https://api.github.com/repos/kpu/kenlm/issues/comments/515284951","html_url":"https://github.com/kpu/kenlm/issues/177#issuecomment-515284951","issue_url":"https://api.github.com/repos/kpu/kenlm/issues/177","id":515284951,"node_id":"MDEyOklzc3VlQ29tbWVudDUxNTI4NDk1MQ==","user":{"login":"nonva","id":2256101,"node_id":"MDQ6VXNlcjIyNTYxMDE=","avatar_url":"https://avatars3.githubusercontent.com/u/2256101?v=4","gravatar_id":"","url":"https://api.github.com/users/nonva","html_url":"https://github.com/nonva","followers_url":"https://api.github.com/users/nonva/followers","following_url":"https://api.github.com/users/nonva/following{/other_user}","gists_url":"https://api.github.com/users/nonva/gists{/gist_id}","starred_url":"https://api.github.com/users/nonva/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nonva/subscriptions","organizations_url":"https://api.github.com/users/nonva/orgs","repos_url":"https://api.github.com/users/nonva/repos","events_url":"https://api.github.com/users/nonva/events{/privacy}","received_events_url":"https://api.github.com/users/nonva/received_events","type":"User","site_admin":false},"created_at":"2019-07-26T02:31:55Z","updated_at":"2019-07-26T02:31:55Z","author_association":"NONE","body":"\r\n\r\n\r\n> @kpu A similar problem I met is like this when I use a 105G corpus with totally default settings.\r\n> command i used: lmplz --prune 0 5 30 -o 3 < corpus.txt > arpa.arpa\r\n> Error:\r\n> === 1/5 Counting and sorting n-grams ===\r\n> Reading /home/user/corpus/corpus-lm/lm_total.txt\r\n> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100\r\n> \r\n> Last input should have been poison. The program should end soon with an error. If it doesn't, there's a bug.\r\n> /home/user/kenlm/util/file.cc:228 in void util::WriteOrThrow(int, const void*, std::size_t) threw FDException because `ret < 1'.\r\n> No space left on device in /tmp/GJTweJ (deleted) while writing 5044497740 bytes\r\n> ______ (__________)\r\n> \r\n> Is that just because the corpus is big so the disk is not enough?\r\n> The size of the file to be written barely changes a little when I use different pruning threshold.\r\n\r\nhow much space needed to train big corpus like 100G？"}}
{"actor":{"display_login":"nonva"}}