{"action":"created","comment":{"url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/comments/178094098","pull_request_review_id":108095388,"id":178094098,"diff_hunk":"@@ -37,11 +49,82 @@ impl BuiltinEntityParser {\n         &self,\n         sentence: &str,\n         filter_entity_kinds: Option<&[BuiltinEntityKind]>,\n+    ) -> Vec<BuiltinEntity> {\n+        if NON_SPACE_SEPARATED_LANGUAGES.contains(&self.lang) {\n+            let original_tokens_bytes_ranges: Vec<Range<usize>> = NON_SPACE_REGEX\n+                .find_iter(sentence)\n+                .map(|m| Range { start: m.start(), end: m.end() })\n+                .collect();\n+\n+            let joined_sentence = original_tokens_bytes_ranges\n+                .iter()\n+                .map(|r| &sentence[r.start..r.end])\n+                .join(\"\");\n+\n+            if original_tokens_bytes_ranges.len() == 0 {\n+                return vec![];\n+            }\n+\n+            let joined_sentence_match_end_byte_index_to_token_index = HashMap::<usize, usize>::from_iter(","path":"snips-nlu-ontology-parsers/src/builtin_entity_parser.rs","position":50,"original_position":50,"commit_id":"0ebe83eee77fe729724b8cab8b0f7a26ee2b2d38","original_commit_id":"0ebe83eee77fe729724b8cab8b0f7a26ee2b2d38","user":{"login":"adrienball","id":11029979,"avatar_url":"https://avatars0.githubusercontent.com/u/11029979?v=4","gravatar_id":"","url":"https://api.github.com/users/adrienball","html_url":"https://github.com/adrienball","followers_url":"https://api.github.com/users/adrienball/followers","following_url":"https://api.github.com/users/adrienball/following{/other_user}","gists_url":"https://api.github.com/users/adrienball/gists{/gist_id}","starred_url":"https://api.github.com/users/adrienball/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adrienball/subscriptions","organizations_url":"https://api.github.com/users/adrienball/orgs","repos_url":"https://api.github.com/users/adrienball/repos","events_url":"https://api.github.com/users/adrienball/events{/privacy}","received_events_url":"https://api.github.com/users/adrienball/received_events","type":"User","site_admin":false},"body":"This is very long, what about `ranges_mapping` with a small comment explaining what it is?","created_at":"2018-03-29T15:29:25Z","updated_at":"2018-03-29T15:29:25Z","html_url":"https://github.com/snipsco/snips-nlu-ontology/pull/32#discussion_r178094098","pull_request_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/32","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/comments/178094098"},"html":{"href":"https://github.com/snipsco/snips-nlu-ontology/pull/32#discussion_r178094098"},"pull_request":{"href":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/32"}}},"pull_request":{"url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/32","id":178346230,"html_url":"https://github.com/snipsco/snips-nlu-ontology/pull/32","diff_url":"https://github.com/snipsco/snips-nlu-ontology/pull/32.diff","patch_url":"https://github.com/snipsco/snips-nlu-ontology/pull/32.patch","issue_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/issues/32","number":32,"state":"open","locked":false,"title":"Fix entity parsing for languages where tokens are not spaces","user":{"login":"ClemDoum","id":4267202,"avatar_url":"https://avatars2.githubusercontent.com/u/4267202?v=4","gravatar_id":"","url":"https://api.github.com/users/ClemDoum","html_url":"https://github.com/ClemDoum","followers_url":"https://api.github.com/users/ClemDoum/followers","following_url":"https://api.github.com/users/ClemDoum/following{/other_user}","gists_url":"https://api.github.com/users/ClemDoum/gists{/gist_id}","starred_url":"https://api.github.com/users/ClemDoum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClemDoum/subscriptions","organizations_url":"https://api.github.com/users/ClemDoum/orgs","repos_url":"https://api.github.com/users/ClemDoum/repos","events_url":"https://api.github.com/users/ClemDoum/events{/privacy}","received_events_url":"https://api.github.com/users/ClemDoum/received_events","type":"User","site_admin":false},"body":"## Bug\r\n\r\nASR outputs tokens with space separators\r\nFor some languages rustling rules expect contiguous chars\r\n\r\nex with a time `the day before chirstmas eve` \r\n-  rustling parsing for `クリスマスイブの前の日`\r\n```\r\n➜  cli git:(develop) cargo run -- --lang ja parse \"クリスマスイブの前の日\"\r\n    Finished dev [unoptimized + debuginfo] target(s) in 1.15 secs\r\n     Running `/Users/davidleroy/Documents/nlp/rustling-ontology/target/debug/rustling-cli --lang ja parse 'クリスマスイブの前の日'`\r\n+----+--------+---+------------------------+-----------------------------------------------------------------------------------------------------+\r\n| ix | log(p) | p | text                   | value                                                                                               |\r\n+====+========+===+========================+=====================================================================================================+\r\n| 0  | 0      | 1 | クリスマスイブの前の日 | Time(TimeOutput { moment: 2018-12-23T00:00:00+01:00, grain: Day, precision: Exact, latent: false }) |\r\n+----+--------+---+------------------------+-----------------------------------------------------------------------------------------------------+\r\n```\r\n\r\n- rustling parsing (wrong parsing - christmas eve only) for the tokenized version of it  `クリスマスイブ(space)の(space)前(space)の(space)日`\r\n```\r\n➜  cli git:(develop) cargo run -- --lang ja parse \"クリスマスイブ の 前 の 日\"\r\n    Finished dev [unoptimized + debuginfo] target(s) in 1.27 secs\r\n     Running `/Users/davidleroy/Documents/nlp/rustling-ontology/target/debug/rustling-cli --lang ja parse 'クリスマスイブ の 前 の 日'`\r\n+----+--------+---+--------------------------------+-----------------------------------------------------------------------------------------------------+\r\n| ix | log(p) | p | text                           | value                                                                                               |\r\n+====+========+===+================================+=====================================================================================================+\r\n| 0  | 0      | 1 | クリスマスイブ________________ | Time(TimeOutput { moment: 2018-12-24T00:00:00+01:00, grain: Day, precision: Exact, latent: false }) |\r\n+----+--------+---+--------------------------------+-----------------------------------------------------------------------------------------------------+\r\n```\r\n\r\n## Solution\r\n\r\nWhen the language is supposed to contain no space:\r\n- we join the tokens separated by tokens\r\n- we parse the sentence\r\n- we remap the ranges of entities to the original ranges\r\n- if we find ranges that do not correspond to original tokens ranges the entity is discarded","created_at":"2018-03-29T14:30:55Z","updated_at":"2018-03-29T15:29:25Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bafc91b5c64737c8a6d170b22757906d2f6f9149","assignee":null,"assignees":[],"requested_reviewers":[{"login":"klefevre","id":1300874,"avatar_url":"https://avatars3.githubusercontent.com/u/1300874?v=4","gravatar_id":"","url":"https://api.github.com/users/klefevre","html_url":"https://github.com/klefevre","followers_url":"https://api.github.com/users/klefevre/followers","following_url":"https://api.github.com/users/klefevre/following{/other_user}","gists_url":"https://api.github.com/users/klefevre/gists{/gist_id}","starred_url":"https://api.github.com/users/klefevre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/klefevre/subscriptions","organizations_url":"https://api.github.com/users/klefevre/orgs","repos_url":"https://api.github.com/users/klefevre/repos","events_url":"https://api.github.com/users/klefevre/events{/privacy}","received_events_url":"https://api.github.com/users/klefevre/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/32/commits","review_comments_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/32/comments","review_comment_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/comments{/number}","comments_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/issues/32/comments","statuses_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/statuses/0ebe83eee77fe729724b8cab8b0f7a26ee2b2d38","head":{"label":"snipsco:fix/non-spaced-languages-parsing","ref":"fix/non-spaced-languages-parsing","sha":"0ebe83eee77fe729724b8cab8b0f7a26ee2b2d38","user":{"login":"snipsco","id":2564618,"avatar_url":"https://avatars0.githubusercontent.com/u/2564618?v=4","gravatar_id":"","url":"https://api.github.com/users/snipsco","html_url":"https://github.com/snipsco","followers_url":"https://api.github.com/users/snipsco/followers","following_url":"https://api.github.com/users/snipsco/following{/other_user}","gists_url":"https://api.github.com/users/snipsco/gists{/gist_id}","starred_url":"https://api.github.com/users/snipsco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/snipsco/subscriptions","organizations_url":"https://api.github.com/users/snipsco/orgs","repos_url":"https://api.github.com/users/snipsco/repos","events_url":"https://api.github.com/users/snipsco/events{/privacy}","received_events_url":"https://api.github.com/users/snipsco/received_events","type":"Organization","site_admin":false},"repo":{"id":121518978,"name":"snips-nlu-ontology","full_name":"snipsco/snips-nlu-ontology","owner":{"login":"snipsco","id":2564618,"avatar_url":"https://avatars0.githubusercontent.com/u/2564618?v=4","gravatar_id":"","url":"https://api.github.com/users/snipsco","html_url":"https://github.com/snipsco","followers_url":"https://api.github.com/users/snipsco/followers","following_url":"https://api.github.com/users/snipsco/following{/other_user}","gists_url":"https://api.github.com/users/snipsco/gists{/gist_id}","starred_url":"https://api.github.com/users/snipsco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/snipsco/subscriptions","organizations_url":"https://api.github.com/users/snipsco/orgs","repos_url":"https://api.github.com/users/snipsco/repos","events_url":"https://api.github.com/users/snipsco/events{/privacy}","received_events_url":"https://api.github.com/users/snipsco/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/snipsco/snips-nlu-ontology","description":"Ontology of the Snips NLU","fork":false,"url":"https://api.github.com/repos/snipsco/snips-nlu-ontology","forks_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/forks","keys_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/keys{/key_id}","collaborators_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/teams","hooks_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/hooks","issue_events_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/issues/events{/number}","events_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/events","assignees_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/assignees{/user}","branches_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/branches{/branch}","tags_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/tags","blobs_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/git/refs{/sha}","trees_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/git/trees{/sha}","statuses_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/statuses/{sha}","languages_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/languages","stargazers_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/stargazers","contributors_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/contributors","subscribers_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/subscribers","subscription_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/subscription","commits_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/commits{/sha}","git_commits_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/git/commits{/sha}","comments_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/comments{/number}","issue_comment_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/issues/comments{/number}","contents_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/contents/{+path}","compare_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/compare/{base}...{head}","merges_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/merges","archive_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/downloads","issues_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/issues{/number}","pulls_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls{/number}","milestones_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/milestones{/number}","notifications_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/labels{/name}","releases_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/releases{/id}","deployments_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/deployments","created_at":"2018-02-14T14:19:29Z","updated_at":"2018-03-19T17:25:45Z","pushed_at":"2018-03-29T14:30:56Z","git_url":"git://github.com/snipsco/snips-nlu-ontology.git","ssh_url":"git@github.com:snipsco/snips-nlu-ontology.git","clone_url":"https://github.com/snipsco/snips-nlu-ontology.git","svn_url":"https://github.com/snipsco/snips-nlu-ontology","homepage":null,"size":214,"stargazers_count":14,"watchers_count":14,"language":"Rust","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":1,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":1,"open_issues":1,"watchers":14,"default_branch":"develop"}},"base":{"label":"snipsco:develop","ref":"develop","sha":"a9762b3ef422a12cfe81a1aaa3f86d6eb95d3dd2","user":{"login":"snipsco","id":2564618,"avatar_url":"https://avatars0.githubusercontent.com/u/2564618?v=4","gravatar_id":"","url":"https://api.github.com/users/snipsco","html_url":"https://github.com/snipsco","followers_url":"https://api.github.com/users/snipsco/followers","following_url":"https://api.github.com/users/snipsco/following{/other_user}","gists_url":"https://api.github.com/users/snipsco/gists{/gist_id}","starred_url":"https://api.github.com/users/snipsco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/snipsco/subscriptions","organizations_url":"https://api.github.com/users/snipsco/orgs","repos_url":"https://api.github.com/users/snipsco/repos","events_url":"https://api.github.com/users/snipsco/events{/privacy}","received_events_url":"https://api.github.com/users/snipsco/received_events","type":"Organization","site_admin":false},"repo":{"id":121518978,"name":"snips-nlu-ontology","full_name":"snipsco/snips-nlu-ontology","owner":{"login":"snipsco","id":2564618,"avatar_url":"https://avatars0.githubusercontent.com/u/2564618?v=4","gravatar_id":"","url":"https://api.github.com/users/snipsco","html_url":"https://github.com/snipsco","followers_url":"https://api.github.com/users/snipsco/followers","following_url":"https://api.github.com/users/snipsco/following{/other_user}","gists_url":"https://api.github.com/users/snipsco/gists{/gist_id}","starred_url":"https://api.github.com/users/snipsco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/snipsco/subscriptions","organizations_url":"https://api.github.com/users/snipsco/orgs","repos_url":"https://api.github.com/users/snipsco/repos","events_url":"https://api.github.com/users/snipsco/events{/privacy}","received_events_url":"https://api.github.com/users/snipsco/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/snipsco/snips-nlu-ontology","description":"Ontology of the Snips NLU","fork":false,"url":"https://api.github.com/repos/snipsco/snips-nlu-ontology","forks_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/forks","keys_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/keys{/key_id}","collaborators_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/teams","hooks_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/hooks","issue_events_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/issues/events{/number}","events_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/events","assignees_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/assignees{/user}","branches_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/branches{/branch}","tags_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/tags","blobs_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/git/refs{/sha}","trees_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/git/trees{/sha}","statuses_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/statuses/{sha}","languages_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/languages","stargazers_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/stargazers","contributors_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/contributors","subscribers_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/subscribers","subscription_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/subscription","commits_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/commits{/sha}","git_commits_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/git/commits{/sha}","comments_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/comments{/number}","issue_comment_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/issues/comments{/number}","contents_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/contents/{+path}","compare_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/compare/{base}...{head}","merges_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/merges","archive_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/downloads","issues_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/issues{/number}","pulls_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls{/number}","milestones_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/milestones{/number}","notifications_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/labels{/name}","releases_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/releases{/id}","deployments_url":"https://api.github.com/repos/snipsco/snips-nlu-ontology/deployments","created_at":"2018-02-14T14:19:29Z","updated_at":"2018-03-19T17:25:45Z","pushed_at":"2018-03-29T14:30:56Z","git_url":"git://github.com/snipsco/snips-nlu-ontology.git","ssh_url":"git@github.com:snipsco/snips-nlu-ontology.git","clone_url":"https://github.com/snipsco/snips-nlu-ontology.git","svn_url":"https://github.com/snipsco/snips-nlu-ontology","homepage":null,"size":214,"stargazers_count":14,"watchers_count":14,"language":"Rust","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":1,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":1,"open_issues":1,"watchers":14,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/32"},"html":{"href":"https://github.com/snipsco/snips-nlu-ontology/pull/32"},"issue":{"href":"https://api.github.com/repos/snipsco/snips-nlu-ontology/issues/32"},"comments":{"href":"https://api.github.com/repos/snipsco/snips-nlu-ontology/issues/32/comments"},"review_comments":{"href":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/32/comments"},"review_comment":{"href":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/snipsco/snips-nlu-ontology/pulls/32/commits"},"statuses":{"href":"https://api.github.com/repos/snipsco/snips-nlu-ontology/statuses/0ebe83eee77fe729724b8cab8b0f7a26ee2b2d38"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"adrienball"}}