{"action":"created","issue":{"url":"https://api.github.com/repos/TextDB/textdb/issues/114","repository_url":"https://api.github.com/repos/TextDB/textdb","labels_url":"https://api.github.com/repos/TextDB/textdb/issues/114/labels{/name}","comments_url":"https://api.github.com/repos/TextDB/textdb/issues/114/comments","events_url":"https://api.github.com/repos/TextDB/textdb/issues/114/events","html_url":"https://github.com/TextDB/textdb/issues/114","id":156195752,"number":114,"title":"DataReader tokenizeQuery problem with Regex","user":{"login":"zuozhi","id":12578068,"avatar_url":"https://avatars.githubusercontent.com/u/12578068?v=3","gravatar_id":"","url":"https://api.github.com/users/zuozhi","html_url":"https://github.com/zuozhi","followers_url":"https://api.github.com/users/zuozhi/followers","following_url":"https://api.github.com/users/zuozhi/following{/other_user}","gists_url":"https://api.github.com/users/zuozhi/gists{/gist_id}","starred_url":"https://api.github.com/users/zuozhi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zuozhi/subscriptions","organizations_url":"https://api.github.com/users/zuozhi/orgs","repos_url":"https://api.github.com/users/zuozhi/repos","events_url":"https://api.github.com/users/zuozhi/events{/privacy}","received_events_url":"https://api.github.com/users/zuozhi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-05-23T03:56:36Z","updated_at":"2016-05-23T05:50:34Z","closed_at":null,"body":"I'm testing RegexMatcher with our new translator, and I encountered a problem with DataReader.\r\nIn DataReader.open(), there's a line calling `tokenizeQuery`. `tokenizeQuery`'s job is to convert a query according to analyzer, for example, `\"new york\"` is converted to `[\"new\", \"york\"]`.\r\n\r\nHowever, regex's query looks like this: `\"abc AND (bcd OR bce)\"`, and regex uses an analyzer with trigram tokenizer. So regex doesn't need to do this. \r\n\r\nSo we need to treat regex query separately. The easiest way to modify DataReader is to add \r\n```\r\nif (isRegex) { \r\n    add some new code\r\n} else { \r\n    keep using current code\r\n}\r\n```\r\n However, a problem is DataReader doesn't know which predicate is using it (and probably isn't supposed to know).\r\n\r\n@chenlica @laisycs @sandeepreddy602 \r\nThoughts about how to solve this problem?"},"comment":{"url":"https://api.github.com/repos/TextDB/textdb/issues/comments/220892527","html_url":"https://github.com/TextDB/textdb/issues/114#issuecomment-220892527","issue_url":"https://api.github.com/repos/TextDB/textdb/issues/114","id":220892527,"user":{"login":"chenlica","id":12926365,"avatar_url":"https://avatars.githubusercontent.com/u/12926365?v=3","gravatar_id":"","url":"https://api.github.com/users/chenlica","html_url":"https://github.com/chenlica","followers_url":"https://api.github.com/users/chenlica/followers","following_url":"https://api.github.com/users/chenlica/following{/other_user}","gists_url":"https://api.github.com/users/chenlica/gists{/gist_id}","starred_url":"https://api.github.com/users/chenlica/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenlica/subscriptions","organizations_url":"https://api.github.com/users/chenlica/orgs","repos_url":"https://api.github.com/users/chenlica/repos","events_url":"https://api.github.com/users/chenlica/events{/privacy}","received_events_url":"https://api.github.com/users/chenlica/received_events","type":"User","site_admin":false},"created_at":"2016-05-23T05:50:33Z","updated_at":"2016-05-23T05:50:33Z","body":"@zuozhi : how about the following.  As an exercise, you implement a new `RegexDataReader` based on your own logic.  After that, we compare it with the existing `DataReader`, and see if we can do some refactoring?"}}
{"actor":{"display_login":"chenlica"}}