{"action":"created","issue":{"url":"https://api.github.com/repos/tmbdev/clstm/issues/106","repository_url":"https://api.github.com/repos/tmbdev/clstm","labels_url":"https://api.github.com/repos/tmbdev/clstm/issues/106/labels{/name}","comments_url":"https://api.github.com/repos/tmbdev/clstm/issues/106/comments","events_url":"https://api.github.com/repos/tmbdev/clstm/issues/106/events","html_url":"https://github.com/tmbdev/clstm/issues/106","id":187620713,"number":106,"title":"load pretrained model assert failed","user":{"login":"striversist","id":3971111,"avatar_url":"https://avatars.githubusercontent.com/u/3971111?v=3","gravatar_id":"","url":"https://api.github.com/users/striversist","html_url":"https://github.com/striversist","followers_url":"https://api.github.com/users/striversist/followers","following_url":"https://api.github.com/users/striversist/following{/other_user}","gists_url":"https://api.github.com/users/striversist/gists{/gist_id}","starred_url":"https://api.github.com/users/striversist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/striversist/subscriptions","organizations_url":"https://api.github.com/users/striversist/orgs","repos_url":"https://api.github.com/users/striversist/repos","events_url":"https://api.github.com/users/striversist/events{/privacy}","received_events_url":"https://api.github.com/users/striversist/received_events","type":"User","site_admin":false},"labels":[{"id":165925611,"url":"https://api.github.com/repos/tmbdev/clstm/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2016-11-07T04:00:43Z","updated_at":"2016-11-09T02:00:07Z","closed_at":null,"body":"Load a pretrained model to retrain new samples will cause assert failed in `Codec::encode`, but start training from scratch, this problem probably not happens.\r\nsee related issue #83 \r\n\r\nAfter digging the code a little, I found this clue:\r\nfrom `clstmocrtrain.cc main1`\r\n```\r\nif (load_name != \"\") {\r\n    clstm.load(load_name);\r\n  } else {\r\n    Codec codec;\r\n    trainingset.getCodec(codec);\r\n    print(\"got\", codec.size(), \"classes\");\r\n```\r\n\r\nIf training from scratch, the `load_name` is empty, so goes to `trainingset.getCodec(codec);`. In this function, the chain `codec.build(gtnames, charsep);` -> `Codec::set` is executed. So the training samples' all codec are inserted into the encoder map.\r\n\r\nIf loading pretrained model to retrain new samples, the `load_name` is not empty, the `clstm.load(load_name); ` loads pretrained codec into encoder map. Next in the `Codec::encode`, if a new sample string contains a new codec(not in the pretrained encoder map), `assert(encoder->count(c) > 0);` fails.\r\n\r\n\r\nHope contributors fix this problem ASAP.\r\n"},"comment":{"url":"https://api.github.com/repos/tmbdev/clstm/issues/comments/259316324","html_url":"https://github.com/tmbdev/clstm/issues/106#issuecomment-259316324","issue_url":"https://api.github.com/repos/tmbdev/clstm/issues/106","id":259316324,"user":{"login":"striversist","id":3971111,"avatar_url":"https://avatars.githubusercontent.com/u/3971111?v=3","gravatar_id":"","url":"https://api.github.com/users/striversist","html_url":"https://github.com/striversist","followers_url":"https://api.github.com/users/striversist/followers","following_url":"https://api.github.com/users/striversist/following{/other_user}","gists_url":"https://api.github.com/users/striversist/gists{/gist_id}","starred_url":"https://api.github.com/users/striversist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/striversist/subscriptions","organizations_url":"https://api.github.com/users/striversist/orgs","repos_url":"https://api.github.com/users/striversist/repos","events_url":"https://api.github.com/users/striversist/events{/privacy}","received_events_url":"https://api.github.com/users/striversist/received_events","type":"User","site_admin":false},"created_at":"2016-11-09T02:00:07Z","updated_at":"2016-11-09T02:00:07Z","body":"Chinese has so many characters, we often train commonly used ones.\r\nBut sometimes that's not enough. We want to add some uncommon characters, So the problem happens.\r\nIf we retrain from scratch, that will take a long long time without a doubt. I think that's the same story with Japanese training."}}
{"actor":{"display_login":"striversist"}}