{"action":"created","comment":{"url":"https://api.github.com/repos/MycroftAI/mimic/pulls/comments/66012188","id":66012188,"diff_hunk":"@@ -408,44 +601,119 @@ int ts_get_stream_size(cst_tokenstream *ts)\n         return 0;\n }\n \n-cst_string private_ts_getc(cst_tokenstream *ts)\n+void private_ts_getc(cst_tokenstream *ts)\n {","path":"src/utils/cst_tokenstream.c","position":361,"original_position":361,"commit_id":"6c3ba03dc4550e8f76aefd6b9f58b7110a0df9b7","original_commit_id":"6c3ba03dc4550e8f76aefd6b9f58b7110a0df9b7","user":{"login":"forslund","id":804543,"avatar_url":"https://avatars.githubusercontent.com/u/804543?v=3","gravatar_id":"","url":"https://api.github.com/users/forslund","html_url":"https://github.com/forslund","followers_url":"https://api.github.com/users/forslund/followers","following_url":"https://api.github.com/users/forslund/following{/other_user}","gists_url":"https://api.github.com/users/forslund/gists{/gist_id}","starred_url":"https://api.github.com/users/forslund/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/forslund/subscriptions","organizations_url":"https://api.github.com/users/forslund/orgs","repos_url":"https://api.github.com/users/forslund/repos","events_url":"https://api.github.com/users/forslund/events{/privacy}","received_events_url":"https://api.github.com/users/forslund/received_events","type":"User","site_admin":false},"body":"This may be outside of the scope of the PR, but I don't see `private_ts_getc()` used anywhere. Can it be removed all together? It seems to expose `internal_ts_get()`but that makes the name quite counter intuitive.","created_at":"2016-06-07T05:32:15Z","updated_at":"2016-06-07T05:32:15Z","html_url":"https://github.com/MycroftAI/mimic/pull/43#discussion_r66012188","pull_request_url":"https://api.github.com/repos/MycroftAI/mimic/pulls/43","_links":{"self":{"href":"https://api.github.com/repos/MycroftAI/mimic/pulls/comments/66012188"},"html":{"href":"https://github.com/MycroftAI/mimic/pull/43#discussion_r66012188"},"pull_request":{"href":"https://api.github.com/repos/MycroftAI/mimic/pulls/43"}}},"pull_request":{"url":"https://api.github.com/repos/MycroftAI/mimic/pulls/43","id":71843357,"html_url":"https://github.com/MycroftAI/mimic/pull/43","diff_url":"https://github.com/MycroftAI/mimic/pull/43.diff","patch_url":"https://github.com/MycroftAI/mimic/pull/43.patch","issue_url":"https://api.github.com/repos/MycroftAI/mimic/issues/43","number":43,"state":"open","locked":false,"title":"Add UTF-8 support to tokenstream","user":{"login":"zeehio","id":75441,"avatar_url":"https://avatars.githubusercontent.com/u/75441?v=3","gravatar_id":"","url":"https://api.github.com/users/zeehio","html_url":"https://github.com/zeehio","followers_url":"https://api.github.com/users/zeehio/followers","following_url":"https://api.github.com/users/zeehio/following{/other_user}","gists_url":"https://api.github.com/users/zeehio/gists{/gist_id}","starred_url":"https://api.github.com/users/zeehio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zeehio/subscriptions","organizations_url":"https://api.github.com/users/zeehio/orgs","repos_url":"https://api.github.com/users/zeehio/repos","events_url":"https://api.github.com/users/zeehio/events{/privacy}","received_events_url":"https://api.github.com/users/zeehio/received_events","type":"User","site_admin":false},"body":"I was working on the wiki to add multilanguage support to mimic. My first frustration was related to the lack of UTF-8 support in the first step of the language analysis: [the tokenizer](https://github.com/MycroftAI/mimic/wiki/Creating-a-new-voice-for-mimic#step-1-tokenizer), so I finally decided to *do the right thing* and try to add UTF-8 support to tokenstreams.\r\n\r\nThis pull request enables the use of UTF-8 sequences to mark word boundaries and punctuation characters. This includes support for Spanish reverse exclamation marks \"¡\" and interrogation marks \"¿\" (these symbols mark in Spanish the beginning of an exclamation or interrogation respectively), as well as the use of the \"DEVANAGARI DANDA\" character (\"।\") as punctuation in indic languages. Additionally, this makes it trivial to support typographic quotation marks.\r\n\r\n\r\nIt is my first pull request with a significant piece of C code, so it may be below mimic desired C coding quality standards. Feel free to make all comments you want.\r\n\r\nMain implementation changes:\r\n- We move from `1 character = 1 byte` to `1 character = either {1, 2, 3, 4} bytes`.\r\n- The tokenstream uses a Look Up Table (LUT) to know if a character is a whitespace, a punctuation mark, etc. The LUT for each character `charclass` required 256 bytes, one for each possible character. Unicode allows for many more characters. so now it starts with 4 tables of 256 bytes each:\r\n    - `charclass` is used with 1 byte characters.\r\n    - `charclass2` is used for 2-byte characters\r\n    - `charclass3` is used for 3-byte characters\r\n    - `charclass4` is used with 4-byte characters.\r\n\r\n`charclass` behaves like the original LUT table. `charclass2` has room for up to 256 `charclass`-like tables inside it, that are allocated only if required. `charclass3` has room for up to 256 `charclass2`-like tables, that are allocated only if required, etc.\r\n\r\nThis approach is (in my opinion) a reasonable tradeoff between:\r\n - A list (slow to look up)\r\n - A huge table for all unicode code points (wastes memory)\r\n - A red-black tree implementation (more coding required)\r\n\r\n\r\nI have added a UTF-8 unit test, to check things are not broken in the future, although more robust tests should be written.\r\n\r\n","created_at":"2016-05-30T01:05:19Z","updated_at":"2016-06-07T05:32:15Z","closed_at":null,"merged_at":null,"merge_commit_sha":"274201ad430a787b1273c8f6d32370450ba5749f","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/MycroftAI/mimic/pulls/43/commits","review_comments_url":"https://api.github.com/repos/MycroftAI/mimic/pulls/43/comments","review_comment_url":"https://api.github.com/repos/MycroftAI/mimic/pulls/comments{/number}","comments_url":"https://api.github.com/repos/MycroftAI/mimic/issues/43/comments","statuses_url":"https://api.github.com/repos/MycroftAI/mimic/statuses/6c3ba03dc4550e8f76aefd6b9f58b7110a0df9b7","head":{"label":"zeehio:tokenstream_utf8","ref":"tokenstream_utf8","sha":"6c3ba03dc4550e8f76aefd6b9f58b7110a0df9b7","user":{"login":"zeehio","id":75441,"avatar_url":"https://avatars.githubusercontent.com/u/75441?v=3","gravatar_id":"","url":"https://api.github.com/users/zeehio","html_url":"https://github.com/zeehio","followers_url":"https://api.github.com/users/zeehio/followers","following_url":"https://api.github.com/users/zeehio/following{/other_user}","gists_url":"https://api.github.com/users/zeehio/gists{/gist_id}","starred_url":"https://api.github.com/users/zeehio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zeehio/subscriptions","organizations_url":"https://api.github.com/users/zeehio/orgs","repos_url":"https://api.github.com/users/zeehio/repos","events_url":"https://api.github.com/users/zeehio/events{/privacy}","received_events_url":"https://api.github.com/users/zeehio/received_events","type":"User","site_admin":false},"repo":{"id":52299357,"name":"mimic","full_name":"zeehio/mimic","owner":{"login":"zeehio","id":75441,"avatar_url":"https://avatars.githubusercontent.com/u/75441?v=3","gravatar_id":"","url":"https://api.github.com/users/zeehio","html_url":"https://github.com/zeehio","followers_url":"https://api.github.com/users/zeehio/followers","following_url":"https://api.github.com/users/zeehio/following{/other_user}","gists_url":"https://api.github.com/users/zeehio/gists{/gist_id}","starred_url":"https://api.github.com/users/zeehio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zeehio/subscriptions","organizations_url":"https://api.github.com/users/zeehio/orgs","repos_url":"https://api.github.com/users/zeehio/repos","events_url":"https://api.github.com/users/zeehio/events{/privacy}","received_events_url":"https://api.github.com/users/zeehio/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/zeehio/mimic","description":"Mycroft's TTS engine, based on CMU's Flite (Festival Lite)","fork":true,"url":"https://api.github.com/repos/zeehio/mimic","forks_url":"https://api.github.com/repos/zeehio/mimic/forks","keys_url":"https://api.github.com/repos/zeehio/mimic/keys{/key_id}","collaborators_url":"https://api.github.com/repos/zeehio/mimic/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/zeehio/mimic/teams","hooks_url":"https://api.github.com/repos/zeehio/mimic/hooks","issue_events_url":"https://api.github.com/repos/zeehio/mimic/issues/events{/number}","events_url":"https://api.github.com/repos/zeehio/mimic/events","assignees_url":"https://api.github.com/repos/zeehio/mimic/assignees{/user}","branches_url":"https://api.github.com/repos/zeehio/mimic/branches{/branch}","tags_url":"https://api.github.com/repos/zeehio/mimic/tags","blobs_url":"https://api.github.com/repos/zeehio/mimic/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/zeehio/mimic/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/zeehio/mimic/git/refs{/sha}","trees_url":"https://api.github.com/repos/zeehio/mimic/git/trees{/sha}","statuses_url":"https://api.github.com/repos/zeehio/mimic/statuses/{sha}","languages_url":"https://api.github.com/repos/zeehio/mimic/languages","stargazers_url":"https://api.github.com/repos/zeehio/mimic/stargazers","contributors_url":"https://api.github.com/repos/zeehio/mimic/contributors","subscribers_url":"https://api.github.com/repos/zeehio/mimic/subscribers","subscription_url":"https://api.github.com/repos/zeehio/mimic/subscription","commits_url":"https://api.github.com/repos/zeehio/mimic/commits{/sha}","git_commits_url":"https://api.github.com/repos/zeehio/mimic/git/commits{/sha}","comments_url":"https://api.github.com/repos/zeehio/mimic/comments{/number}","issue_comment_url":"https://api.github.com/repos/zeehio/mimic/issues/comments{/number}","contents_url":"https://api.github.com/repos/zeehio/mimic/contents/{+path}","compare_url":"https://api.github.com/repos/zeehio/mimic/compare/{base}...{head}","merges_url":"https://api.github.com/repos/zeehio/mimic/merges","archive_url":"https://api.github.com/repos/zeehio/mimic/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/zeehio/mimic/downloads","issues_url":"https://api.github.com/repos/zeehio/mimic/issues{/number}","pulls_url":"https://api.github.com/repos/zeehio/mimic/pulls{/number}","milestones_url":"https://api.github.com/repos/zeehio/mimic/milestones{/number}","notifications_url":"https://api.github.com/repos/zeehio/mimic/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/zeehio/mimic/labels{/name}","releases_url":"https://api.github.com/repos/zeehio/mimic/releases{/id}","deployments_url":"https://api.github.com/repos/zeehio/mimic/deployments","created_at":"2016-02-22T19:27:18Z","updated_at":"2016-02-22T19:27:35Z","pushed_at":"2016-05-30T01:32:28Z","git_url":"git://github.com/zeehio/mimic.git","ssh_url":"git@github.com:zeehio/mimic.git","clone_url":"https://github.com/zeehio/mimic.git","svn_url":"https://github.com/zeehio/mimic","homepage":"https://mimic.mycroft.ai","size":556052,"stargazers_count":0,"watchers_count":0,"language":"C","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"MycroftAI:development","ref":"development","sha":"c8893c55992580d39ec80c0fb7a691380773dce3","user":{"login":"MycroftAI","id":14171097,"avatar_url":"https://avatars.githubusercontent.com/u/14171097?v=3","gravatar_id":"","url":"https://api.github.com/users/MycroftAI","html_url":"https://github.com/MycroftAI","followers_url":"https://api.github.com/users/MycroftAI/followers","following_url":"https://api.github.com/users/MycroftAI/following{/other_user}","gists_url":"https://api.github.com/users/MycroftAI/gists{/gist_id}","starred_url":"https://api.github.com/users/MycroftAI/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MycroftAI/subscriptions","organizations_url":"https://api.github.com/users/MycroftAI/orgs","repos_url":"https://api.github.com/users/MycroftAI/repos","events_url":"https://api.github.com/users/MycroftAI/events{/privacy}","received_events_url":"https://api.github.com/users/MycroftAI/received_events","type":"Organization","site_admin":false},"repo":{"id":51099127,"name":"mimic","full_name":"MycroftAI/mimic","owner":{"login":"MycroftAI","id":14171097,"avatar_url":"https://avatars.githubusercontent.com/u/14171097?v=3","gravatar_id":"","url":"https://api.github.com/users/MycroftAI","html_url":"https://github.com/MycroftAI","followers_url":"https://api.github.com/users/MycroftAI/followers","following_url":"https://api.github.com/users/MycroftAI/following{/other_user}","gists_url":"https://api.github.com/users/MycroftAI/gists{/gist_id}","starred_url":"https://api.github.com/users/MycroftAI/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MycroftAI/subscriptions","organizations_url":"https://api.github.com/users/MycroftAI/orgs","repos_url":"https://api.github.com/users/MycroftAI/repos","events_url":"https://api.github.com/users/MycroftAI/events{/privacy}","received_events_url":"https://api.github.com/users/MycroftAI/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/MycroftAI/mimic","description":"Mycroft's TTS engine, based on CMU's Flite (Festival Lite)","fork":false,"url":"https://api.github.com/repos/MycroftAI/mimic","forks_url":"https://api.github.com/repos/MycroftAI/mimic/forks","keys_url":"https://api.github.com/repos/MycroftAI/mimic/keys{/key_id}","collaborators_url":"https://api.github.com/repos/MycroftAI/mimic/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/MycroftAI/mimic/teams","hooks_url":"https://api.github.com/repos/MycroftAI/mimic/hooks","issue_events_url":"https://api.github.com/repos/MycroftAI/mimic/issues/events{/number}","events_url":"https://api.github.com/repos/MycroftAI/mimic/events","assignees_url":"https://api.github.com/repos/MycroftAI/mimic/assignees{/user}","branches_url":"https://api.github.com/repos/MycroftAI/mimic/branches{/branch}","tags_url":"https://api.github.com/repos/MycroftAI/mimic/tags","blobs_url":"https://api.github.com/repos/MycroftAI/mimic/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/MycroftAI/mimic/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/MycroftAI/mimic/git/refs{/sha}","trees_url":"https://api.github.com/repos/MycroftAI/mimic/git/trees{/sha}","statuses_url":"https://api.github.com/repos/MycroftAI/mimic/statuses/{sha}","languages_url":"https://api.github.com/repos/MycroftAI/mimic/languages","stargazers_url":"https://api.github.com/repos/MycroftAI/mimic/stargazers","contributors_url":"https://api.github.com/repos/MycroftAI/mimic/contributors","subscribers_url":"https://api.github.com/repos/MycroftAI/mimic/subscribers","subscription_url":"https://api.github.com/repos/MycroftAI/mimic/subscription","commits_url":"https://api.github.com/repos/MycroftAI/mimic/commits{/sha}","git_commits_url":"https://api.github.com/repos/MycroftAI/mimic/git/commits{/sha}","comments_url":"https://api.github.com/repos/MycroftAI/mimic/comments{/number}","issue_comment_url":"https://api.github.com/repos/MycroftAI/mimic/issues/comments{/number}","contents_url":"https://api.github.com/repos/MycroftAI/mimic/contents/{+path}","compare_url":"https://api.github.com/repos/MycroftAI/mimic/compare/{base}...{head}","merges_url":"https://api.github.com/repos/MycroftAI/mimic/merges","archive_url":"https://api.github.com/repos/MycroftAI/mimic/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/MycroftAI/mimic/downloads","issues_url":"https://api.github.com/repos/MycroftAI/mimic/issues{/number}","pulls_url":"https://api.github.com/repos/MycroftAI/mimic/pulls{/number}","milestones_url":"https://api.github.com/repos/MycroftAI/mimic/milestones{/number}","notifications_url":"https://api.github.com/repos/MycroftAI/mimic/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/MycroftAI/mimic/labels{/name}","releases_url":"https://api.github.com/repos/MycroftAI/mimic/releases{/id}","deployments_url":"https://api.github.com/repos/MycroftAI/mimic/deployments","created_at":"2016-02-04T19:18:28Z","updated_at":"2016-06-04T19:21:07Z","pushed_at":"2016-06-01T20:37:08Z","git_url":"git://github.com/MycroftAI/mimic.git","ssh_url":"git@github.com:MycroftAI/mimic.git","clone_url":"https://github.com/MycroftAI/mimic.git","svn_url":"https://github.com/MycroftAI/mimic","homepage":"https://mimic.mycroft.ai","size":557224,"stargazers_count":115,"watchers_count":115,"language":"C","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":12,"mirror_url":null,"open_issues_count":19,"forks":12,"open_issues":19,"watchers":115,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/MycroftAI/mimic/pulls/43"},"html":{"href":"https://github.com/MycroftAI/mimic/pull/43"},"issue":{"href":"https://api.github.com/repos/MycroftAI/mimic/issues/43"},"comments":{"href":"https://api.github.com/repos/MycroftAI/mimic/issues/43/comments"},"review_comments":{"href":"https://api.github.com/repos/MycroftAI/mimic/pulls/43/comments"},"review_comment":{"href":"https://api.github.com/repos/MycroftAI/mimic/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/MycroftAI/mimic/pulls/43/commits"},"statuses":{"href":"https://api.github.com/repos/MycroftAI/mimic/statuses/6c3ba03dc4550e8f76aefd6b9f58b7110a0df9b7"}}}}
{"actor":{"display_login":"forslund"}}