{"action":"created","issue":{"url":"https://api.github.com/repos/OpenNMT/OpenNMT/issues/25","repository_url":"https://api.github.com/repos/OpenNMT/OpenNMT","labels_url":"https://api.github.com/repos/OpenNMT/OpenNMT/issues/25/labels{/name}","comments_url":"https://api.github.com/repos/OpenNMT/OpenNMT/issues/25/comments","events_url":"https://api.github.com/repos/OpenNMT/OpenNMT/issues/25/events","html_url":"https://github.com/OpenNMT/OpenNMT/issues/25","id":197171487,"number":25,"title":"All words must have the same number of Features","user":{"login":"mdasadul","id":8009589,"avatar_url":"https://avatars.githubusercontent.com/u/8009589?v=3","gravatar_id":"","url":"https://api.github.com/users/mdasadul","html_url":"https://github.com/mdasadul","followers_url":"https://api.github.com/users/mdasadul/followers","following_url":"https://api.github.com/users/mdasadul/following{/other_user}","gists_url":"https://api.github.com/users/mdasadul/gists{/gist_id}","starred_url":"https://api.github.com/users/mdasadul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdasadul/subscriptions","organizations_url":"https://api.github.com/users/mdasadul/orgs","repos_url":"https://api.github.com/users/mdasadul/repos","events_url":"https://api.github.com/users/mdasadul/events{/privacy}","received_events_url":"https://api.github.com/users/mdasadul/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-12-22T13:10:56Z","updated_at":"2016-12-22T13:55:16Z","closed_at":null,"body":"I am trying to perform preprocessing  with different data than the demo and ended up getting following errors \r\n\r\n/usr/bin/luajit: ./onmt/utils/Features.lua:17: all words must have the same number of features\r\nstack traceback:\r\n        [C]: in function 'assert'\r\n        ./onmt/utils/Features.lua:17: in function 'extract'\r\n        preprocess.lua:56: in function 'makeVocabulary'\r\n        preprocess.lua:122: in function 'initVocabulary'\r\n        preprocess.lua:266: in function 'main'\r\n        preprocess.lua:301: in main chunk\r\n        [C]: in function 'dofile'\r\n        /usr/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk\r\n        [C]: at 0x00406670\r\n\r\nAlso I am wondering is there will be any preprocessor-shards.py here as data size grows few GB fitting them into memory will not be possible if we use preprocessing.py?"},"comment":{"url":"https://api.github.com/repos/OpenNMT/OpenNMT/issues/comments/268804144","html_url":"https://github.com/OpenNMT/OpenNMT/issues/25#issuecomment-268804144","issue_url":"https://api.github.com/repos/OpenNMT/OpenNMT/issues/25","id":268804144,"user":{"login":"mdasadul","id":8009589,"avatar_url":"https://avatars.githubusercontent.com/u/8009589?v=3","gravatar_id":"","url":"https://api.github.com/users/mdasadul","html_url":"https://github.com/mdasadul","followers_url":"https://api.github.com/users/mdasadul/followers","following_url":"https://api.github.com/users/mdasadul/following{/other_user}","gists_url":"https://api.github.com/users/mdasadul/gists{/gist_id}","starred_url":"https://api.github.com/users/mdasadul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdasadul/subscriptions","organizations_url":"https://api.github.com/users/mdasadul/orgs","repos_url":"https://api.github.com/users/mdasadul/repos","events_url":"https://api.github.com/users/mdasadul/events{/privacy}","received_events_url":"https://api.github.com/users/mdasadul/received_events","type":"User","site_admin":false},"created_at":"2016-12-22T13:55:15Z","updated_at":"2016-12-22T13:55:15Z","body":"This is the line which is responsible for the error\r\n\\right\\|_0}$ is the number of nonzeroelements.\r\n\r\nI am planning to train with 20M sentences  and I think this will be difficult to fit into 8GB GPU without using shards  "}}
{"actor":{"display_login":"mdasadul"}}