{"action":"closed","issue":{"url":"https://api.github.com/repos/kbenoit/quanteda/issues/287","repository_url":"https://api.github.com/repos/kbenoit/quanteda","labels_url":"https://api.github.com/repos/kbenoit/quanteda/issues/287/labels{/name}","comments_url":"https://api.github.com/repos/kbenoit/quanteda/issues/287/comments","events_url":"https://api.github.com/repos/kbenoit/quanteda/issues/287/events","html_url":"https://github.com/kbenoit/quanteda/issues/287","id":188089357,"number":287,"title":"Too many lapply in tokens generator","user":{"login":"koheiw","id":6572963,"avatar_url":"https://avatars.githubusercontent.com/u/6572963?v=3","gravatar_id":"","url":"https://api.github.com/users/koheiw","html_url":"https://github.com/koheiw","followers_url":"https://api.github.com/users/koheiw/followers","following_url":"https://api.github.com/users/koheiw/following{/other_user}","gists_url":"https://api.github.com/users/koheiw/gists{/gist_id}","starred_url":"https://api.github.com/users/koheiw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koheiw/subscriptions","organizations_url":"https://api.github.com/users/koheiw/orgs","repos_url":"https://api.github.com/users/koheiw/repos","events_url":"https://api.github.com/users/koheiw/events{/privacy}","received_events_url":"https://api.github.com/users/koheiw/received_events","type":"User","site_admin":false},"labels":[{"id":476260765,"url":"https://api.github.com/repos/kbenoit/quanteda/labels/performance","name":"performance","color":"0052cc","default":false},{"id":476279735,"url":"https://api.github.com/repos/kbenoit/quanteda/labels/tokenizing","name":"tokenizing","color":"e99695","default":false}],"state":"closed","locked":false,"assignee":{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars.githubusercontent.com/u/2182246?v=3","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},"assignees":[{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars.githubusercontent.com/u/2182246?v=3","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},{"login":"koheiw","id":6572963,"avatar_url":"https://avatars.githubusercontent.com/u/6572963?v=3","gravatar_id":"","url":"https://api.github.com/users/koheiw","html_url":"https://github.com/koheiw","followers_url":"https://api.github.com/users/koheiw/followers","following_url":"https://api.github.com/users/koheiw/following{/other_user}","gists_url":"https://api.github.com/users/koheiw/gists{/gist_id}","starred_url":"https://api.github.com/users/koheiw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koheiw/subscriptions","organizations_url":"https://api.github.com/users/koheiw/orgs","repos_url":"https://api.github.com/users/koheiw/repos","events_url":"https://api.github.com/users/koheiw/events{/privacy}","received_events_url":"https://api.github.com/users/koheiw/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kbenoit/quanteda/milestones/2","html_url":"https://github.com/kbenoit/quanteda/milestone/2","labels_url":"https://api.github.com/repos/kbenoit/quanteda/milestones/2/labels","id":2221126,"number":2,"title":"v0.9.9 release","description":"","creator":{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars.githubusercontent.com/u/2182246?v=3","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},"open_issues":22,"closed_issues":8,"state":"open","created_at":"2016-12-30T12:08:44Z","updated_at":"2016-12-30T16:57:54Z","due_on":"2017-01-08T08:00:00Z","closed_at":null},"comments":4,"created_at":"2016-11-08T20:02:05Z","updated_at":"2016-12-30T16:57:54Z","closed_at":"2016-12-30T16:57:54Z","body":"R's lapply function is slow compared to R's vectorized operations or C++ loops. But we are using many lapply in the tokens generator. One solution is to use C++ function for simple cleaning like ``result <- qatd_cpp_remove_string_list(result, \"\")`` (I used this already and tokenization became 30% faster). \r\n\r\nIf the operation is not so simple, we can put lapply together. For example,\r\n```\r\nresult <- lapply(result, function(x){ \r\n      x <- stringi::stri_replace_all_charclass(x, \"[\\\\p{S}]\", \"\")\r\n      x[which(x != \"\")]\r\n})\r\n```\r\ninstead of \r\n```\r\nresult <- lapply(result, stringi::stri_replace_all_charclass, \"[\\\\p{S}]\", \"\")\r\nresult <- lapply(result, function(x) x <- x[which(x != \"\")])\r\n```\r\nThis is basically why our tokenizer is slower than ``tokenizers``.\r\n"}}
{"actor":{"display_login":"koheiw"}}