{"action":"opened","issue":{"url":"https://api.github.com/repos/Esukhia/pybo/issues/18","repository_url":"https://api.github.com/repos/Esukhia/pybo","labels_url":"https://api.github.com/repos/Esukhia/pybo/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/Esukhia/pybo/issues/18/comments","events_url":"https://api.github.com/repos/Esukhia/pybo/issues/18/events","html_url":"https://github.com/Esukhia/pybo/issues/18","id":319453492,"number":18,"title":"tokenizer bug","user":{"login":"mikkokotila","id":7943188,"avatar_url":"https://avatars1.githubusercontent.com/u/7943188?v=4","gravatar_id":"","url":"https://api.github.com/users/mikkokotila","html_url":"https://github.com/mikkokotila","followers_url":"https://api.github.com/users/mikkokotila/followers","following_url":"https://api.github.com/users/mikkokotila/following{/other_user}","gists_url":"https://api.github.com/users/mikkokotila/gists{/gist_id}","starred_url":"https://api.github.com/users/mikkokotila/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikkokotila/subscriptions","organizations_url":"https://api.github.com/users/mikkokotila/orgs","repos_url":"https://api.github.com/users/mikkokotila/repos","events_url":"https://api.github.com/users/mikkokotila/events{/privacy}","received_events_url":"https://api.github.com/users/mikkokotila/received_events","type":"User","site_admin":false},"labels":[{"id":831495918,"url":"https://api.github.com/repos/Esukhia/pybo/labels/bug","name":"bug","color":"d73a4a","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-02T08:15:00Z","updated_at":"2018-05-02T08:15:00Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"The below comes up for one volume in Rinchen Terdzo, in a scan of the whole body of texts. I tried to manually reproduce but could not. \r\n\r\nThe line that is causing it seems to be: \r\n\r\n`tokens.append(self.chunks_to_token([c_idx]))`\r\n\r\nin tokenizer.py. \r\n\r\nThe trace shows that this does not appear to be the same issue with #8.\r\n\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nIndexError                                Traceback (most recent call last)\r\n<ipython-input-221-9b0503e8f2ce> in <module>()\r\n----> 1 word2vec_pipebig('rt_raw_clean.txt')\r\n\r\n<ipython-input-218-52a921d05e0c> in word2vec_pipebig(filename, save_model)\r\n    196                 model = word2vec_pipeline(line)\r\n    197             else:\r\n--> 198                 model = word2vec_pipeline(line, build_model=model)\r\n    199             x += 1\r\n    200 \r\n\r\n<ipython-input-218-52a921d05e0c> in word2vec_pipeline(docs, epochs, skipgrams, workers, save, from_file, build_model)\r\n    136         docs = read_file(docs)\r\n    137 \r\n--> 138     tokens = tokenize(docs)\r\n    139     sentences = word2vec_prep(tokens)\r\n    140 \r\n\r\n<ipython-input-218-52a921d05e0c> in tokenize(text)\r\n     61             pass\r\n     62 \r\n---> 63     return tok.tokenize(out, split_affixes=False)\r\n     64 \r\n     65 \r\n\r\n~/dev/astetik_test/lib/python3.6/site-packages/pybo/__init__.py in tokenize(self, string, split_affixes)\r\n     50         \"\"\"\r\n     51         preprocessed = PyBoTextChunks(string)\r\n---> 52         tokens = self.tok.tokenize(preprocessed, split_affixes=split_affixes)\r\n     53         if self.lemmatize:\r\n     54             LemmatizeTokens().lemmatize(tokens)\r\n\r\n~/dev/astetik_test/lib/python3.6/site-packages/pybo/tokenizer.py in tokenize(self, pre_processed, split_affixes, debug)\r\n    135                     current_node = None\r\n    136 \r\n--> 137                 tokens.append(self.chunks_to_token([c_idx]))\r\n    138 \r\n    139             # END OF INPUT\r\n\r\n~/dev/astetik_test/lib/python3.6/site-packages/pybo/tokenizer.py in chunks_to_token(self, syls, tag, ttype)\r\n    180         if len(syls) == 1:\r\n    181             # chunk format: ([char_idx1, char_idx2, ...], (type, start_idx, len_idx))\r\n--> 182             token_syls = [self.pre_processed.chunks[syls[0]][0]]\r\n    183             token_type = self.pre_processed.chunks[syls[0]][1][0]\r\n    184             token_start = self.pre_processed.chunks[syls[0]][1][1]\r\n\r\nIndexError: list index out of range\r\n```"}}
{"actor":{"display_login":"mikkokotila"}}