{"action":"created","issue":{"url":"https://api.github.com/repos/groue/GRDB.swift/issues/413","repository_url":"https://api.github.com/repos/groue/GRDB.swift","labels_url":"https://api.github.com/repos/groue/GRDB.swift/issues/413/labels{/name}","comments_url":"https://api.github.com/repos/groue/GRDB.swift/issues/413/comments","events_url":"https://api.github.com/repos/groue/GRDB.swift/issues/413/events","html_url":"https://github.com/groue/GRDB.swift/issues/413","id":362460576,"node_id":"MDU6SXNzdWUzNjI0NjA1NzY=","number":413,"title":"How can I use FTS5 Tokenizers to search Chinese ?","user":{"login":"qiulang","id":516243,"node_id":"MDQ6VXNlcjUxNjI0Mw==","avatar_url":"https://avatars0.githubusercontent.com/u/516243?v=4","gravatar_id":"","url":"https://api.github.com/users/qiulang","html_url":"https://github.com/qiulang","followers_url":"https://api.github.com/users/qiulang/followers","following_url":"https://api.github.com/users/qiulang/following{/other_user}","gists_url":"https://api.github.com/users/qiulang/gists{/gist_id}","starred_url":"https://api.github.com/users/qiulang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qiulang/subscriptions","organizations_url":"https://api.github.com/users/qiulang/orgs","repos_url":"https://api.github.com/users/qiulang/repos","events_url":"https://api.github.com/users/qiulang/events{/privacy}","received_events_url":"https://api.github.com/users/qiulang/received_events","type":"User","site_admin":false},"labels":[{"id":230659300,"node_id":"MDU6TGFiZWwyMzA2NTkzMDA=","url":"https://api.github.com/repos/groue/GRDB.swift/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-09-21T05:29:26Z","updated_at":"2018-09-22T03:52:25Z","closed_at":null,"author_association":"NONE","body":"### What did you do?\r\n\r\nI have carefully read your FTS5 Tokenizers document and think maybe I can use a good Chinese Word Segmentation library, e.g https://github.com/yanyiwu/cppjieba/blob/master/README_EN.md, to do a better FTS in sqlite. But I have faced some problems.\r\n\r\nFirst of all, have you noticed FTS5 Unicode61 Tokenizer seems not support CJK at all!\r\nI asked the question at [SO](https://stackoverflow.com/questions/52422437/why-sqlite-fts5-unicode61-tokenizer-does-not-support-cjkchinese-japanese-krean) and [sqlite mail list](https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg112010.html)\r\n\r\nNo answer yet. \r\n\r\nSo if FTS5 Unicode61 Tokenizer does indeed **not** support Chinese then I doubt if I use your **FTS5WrapperTokenizer** with `unicode61` as `wrappedTokenizer` to work for Chinese?\r\n\r\nI tried to build a sqlite **icu** version but failed. All the documents I googled seem outdated.  So I did not try your FTS3 tokenizer. I don't know if sqlite does not support icu can fts still work for CJK ?  I have limited knowledge about sqlite in this area so correct me if I was wrong.\r\n\r\nSecond, say I find a fts5 tokenizer to tokenize Chinese characters one by one, then I can use https://github.com/yanyiwu/cppjieba/ to further segment Chinese words. To use your FTS5WrapperTokenizer, I have to implement the `accept(token:flags:for:tokenCallback:)` method can call cppjieba, right ? And since it is c++ I have to wrap it swift \r\n\r\nOr I implement FTS5CustomTokenizer to call cppjieba directly. Which one is easier from your point of view?\r\n\r\nThanks!\r\n\r\nQiulang\r\n\r\n### Environment\r\n\r\n**GRDB flavor(s):** GRDB\r\n**GRDB version:** latest\r\n**Xcode version:** 10\r\n**Swift version:** 4\r\n**Platform(s) running GRDB:**macOS\r\n**macOS version running Xcode:** 10.13\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/groue/GRDB.swift/issues/comments/423714241","html_url":"https://github.com/groue/GRDB.swift/issues/413#issuecomment-423714241","issue_url":"https://api.github.com/repos/groue/GRDB.swift/issues/413","id":423714241,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzcxNDI0MQ==","user":{"login":"qiulang","id":516243,"node_id":"MDQ6VXNlcjUxNjI0Mw==","avatar_url":"https://avatars0.githubusercontent.com/u/516243?v=4","gravatar_id":"","url":"https://api.github.com/users/qiulang","html_url":"https://github.com/qiulang","followers_url":"https://api.github.com/users/qiulang/followers","following_url":"https://api.github.com/users/qiulang/following{/other_user}","gists_url":"https://api.github.com/users/qiulang/gists{/gist_id}","starred_url":"https://api.github.com/users/qiulang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qiulang/subscriptions","organizations_url":"https://api.github.com/users/qiulang/orgs","repos_url":"https://api.github.com/users/qiulang/repos","events_url":"https://api.github.com/users/qiulang/events{/privacy}","received_events_url":"https://api.github.com/users/qiulang/received_events","type":"User","site_admin":false},"created_at":"2018-09-22T03:52:25Z","updated_at":"2018-09-22T03:52:25Z","author_association":"NONE","body":"Yes I will share my whatever I find!"}}
{"actor":{"display_login":"qiulang"}}