title: Wrong tokenization of Japanese verbs using SQLite FTS ICU tokenizer
body: <p>I'm trying to use SQLite FTS ICU tokenizer for Japanese on an Android device.</p>
  <p>Here is my query:</p>
  <p><code>CREATE VIRTUAL TABLE IF NOT EXISTS tokenizer_ja USING fts3tokenize(icu, ja);</code></p>
  <p><code>SELECT token, position, start, end FROM tokenizer_ja WHERE input=?;</code></p>
  <p><code>?</code> is replaced by the phrase I want to tokenize.</p>
  <p>It's almost working, but there is a problem with verbs. For example, the sentence <code>日本語で話します</code> is tokenized as <code>日本語</code>, <code>で</code>, <code>話</code>, <code>し</code>, <code>ます</code>.</p>
  <p><code>話します</code> is one word and should be one token.</p>
  <p>I tried different verbs and there is the same problem with all verbs. Only the dictionary form tokenization is correct.</p>
  <p>Is there a way to have a good tokenization? A tokenizer option that can be activated? Any explanation why the tokenization is like that?</p>