{"action":"created","issue":{"url":"https://api.github.com/repos/quanteda/quanteda/issues/1221","repository_url":"https://api.github.com/repos/quanteda/quanteda","labels_url":"https://api.github.com/repos/quanteda/quanteda/issues/1221/labels{/name}","comments_url":"https://api.github.com/repos/quanteda/quanteda/issues/1221/comments","events_url":"https://api.github.com/repos/quanteda/quanteda/issues/1221/events","html_url":"https://github.com/quanteda/quanteda/issues/1221","id":295859102,"number":1221,"title":"tokens(x, what = \"fasterword\", remove_separators = TRUE) does not remove spaces","user":{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars2.githubusercontent.com/u/2182246?v=4","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},"labels":[{"id":11645690,"url":"https://api.github.com/repos/quanteda/quanteda/labels/bug","name":"bug","color":"fc2929","default":true},{"id":476279735,"url":"https://api.github.com/repos/quanteda/quanteda/labels/tokens","name":"tokens","color":"e99695","default":false}],"state":"open","locked":false,"assignee":{"login":"koheiw","id":6572963,"avatar_url":"https://avatars2.githubusercontent.com/u/6572963?v=4","gravatar_id":"","url":"https://api.github.com/users/koheiw","html_url":"https://github.com/koheiw","followers_url":"https://api.github.com/users/koheiw/followers","following_url":"https://api.github.com/users/koheiw/following{/other_user}","gists_url":"https://api.github.com/users/koheiw/gists{/gist_id}","starred_url":"https://api.github.com/users/koheiw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koheiw/subscriptions","organizations_url":"https://api.github.com/users/koheiw/orgs","repos_url":"https://api.github.com/users/koheiw/repos","events_url":"https://api.github.com/users/koheiw/events{/privacy}","received_events_url":"https://api.github.com/users/koheiw/received_events","type":"User","site_admin":false},"assignees":[{"login":"koheiw","id":6572963,"avatar_url":"https://avatars2.githubusercontent.com/u/6572963?v=4","gravatar_id":"","url":"https://api.github.com/users/koheiw","html_url":"https://github.com/koheiw","followers_url":"https://api.github.com/users/koheiw/followers","following_url":"https://api.github.com/users/koheiw/following{/other_user}","gists_url":"https://api.github.com/users/koheiw/gists{/gist_id}","starred_url":"https://api.github.com/users/koheiw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koheiw/subscriptions","organizations_url":"https://api.github.com/users/koheiw/orgs","repos_url":"https://api.github.com/users/koheiw/repos","events_url":"https://api.github.com/users/koheiw/events{/privacy}","received_events_url":"https://api.github.com/users/koheiw/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-02-09T12:45:10Z","updated_at":"2018-02-11T16:54:56Z","closed_at":null,"author_association":"COLLABORATOR","body":"### `what = \"word\"`\r\n\r\nThis is OK and is what we would expect:\r\n```r\r\ntxt <- \"\\n \\t space\"\r\ntokens(txt, what = \"word\", remove_separators = TRUE)  # OK\r\n# tokens from 1 document.\r\n# text1 :\r\n# [1] \"space\"\r\ntokens(txt, what = \"word\", remove_separators = FALSE) # OK\r\n# tokens from 1 document.\r\n# text1 :\r\n# [1] \"\\n\"    \" \"     \"\\t\"    \" \"     \"space\"\r\n```\r\n\r\n### `what = \"fasterword\"`\r\n\r\nThere are two problems:\r\n1. `remove_separators` seems to have no effect\r\n2. the whitespace characters are converted to `\"\"` and not kept as `\"\\n\"` `\"\\t\"`\r\n```r\r\ntokens(txt, what = \"fasterword\", remove_separators = TRUE)  # NOT OK\r\n# tokens from 1 document.\r\n# text1 :\r\n# [1] \"\"      \"\"      \"\"      \"\"      \"space\"\r\ntokens(txt, what = \"fasterword\", remove_separators = FALSE)  # NOT OK - SHOULD KEEP \\n \\t\r\n# tokens from 1 document.\r\n# text1 :\r\n# [1] \"\"      \"\"      \"\"      \"\"      \"space\"\r\n```\r\nHowever, the second issue appears to be built in to **stringi**, where all matches are returned as `\"\"` unless they are a single space in between tokens:\r\n```r\r\n> stringi::stri_split_charclass(\"test test\", \"\\\\p{WHITE_SPACE}\")\r\n[[1]]\r\n[1] \"test\" \"test\"\r\n\r\n> stringi::stri_split_charclass(\"test \", \"\\\\p{WHITE_SPACE}\")\r\n[[1]]\r\n[1] \"test\" \"\"    \r\n\r\n> stringi::stri_split_charclass(\" \", \"\\\\p{WHITE_SPACE}\")\r\n[[1]]\r\n[1] \"\" \"\"\r\n\r\nstringi::stri_split_charclass(\"test  test\", \"\\\\p{WHITE_SPACE}\")\r\n[[1]]\r\n[1] \"test\" \"\"     \"test\"\r\n```\r\n\r\n### `what = \"fastestword\"`\r\n\r\nThis is OK, since the definition of this type is to match only on spaces:\r\n```r\r\ntokens(txt, what = \"fastestword\", remove_separators = TRUE)  # OK\r\n# tokens from 1 document.\r\n# text1 :\r\n# [1] \"\\n\"    \"\\t\"    \"space\"\r\n```\r\nbut it fails when `remove_separators = FALSE`, because it should return the spaces as well:\r\n```r\r\ntokens(txt, what = \"fastestword\", remove_separators = FALSE)  # NOT OK\r\n# tokens from 1 document.\r\n# text1 :\r\n# [1] \"\\n\"    \"\\t\"    \"space\"\r\n```"},"comment":{"url":"https://api.github.com/repos/quanteda/quanteda/issues/comments/364766741","html_url":"https://github.com/quanteda/quanteda/issues/1221#issuecomment-364766741","issue_url":"https://api.github.com/repos/quanteda/quanteda/issues/1221","id":364766741,"user":{"login":"koheiw","id":6572963,"avatar_url":"https://avatars2.githubusercontent.com/u/6572963?v=4","gravatar_id":"","url":"https://api.github.com/users/koheiw","html_url":"https://github.com/koheiw","followers_url":"https://api.github.com/users/koheiw/followers","following_url":"https://api.github.com/users/koheiw/following{/other_user}","gists_url":"https://api.github.com/users/koheiw/gists{/gist_id}","starred_url":"https://api.github.com/users/koheiw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koheiw/subscriptions","organizations_url":"https://api.github.com/users/koheiw/orgs","repos_url":"https://api.github.com/users/koheiw/repos","events_url":"https://api.github.com/users/koheiw/events{/privacy}","received_events_url":"https://api.github.com/users/koheiw/received_events","type":"User","site_admin":false},"created_at":"2018-02-11T16:54:56Z","updated_at":"2018-02-11T16:54:56Z","author_association":"COLLABORATOR","body":"`stri_detect_regex()` is a bit slower but we have `fastestword` for speed.\r\n```\r\nUnit: microseconds\r\n                                                     expr      min       lq     mean   median       uq      max neval\r\n stringi::stri_split_charclass(txt, \"\\\\\\\\p{WHITE_SPACE}\")  704.386  930.668 1111.130  982.357 1174.155 4817.257   100\r\n              stringi::stri_split_regex(txt, \"\\\\\\\\p{Z}+\") 1275.793 1401.102 1695.516 1521.536 1778.677 4785.205   100\r\n```\r\n"}}
{"actor":{"display_login":"koheiw"}}