{"action":"created","issue":{"url":"https://api.github.com/repos/prestodb/RPresto/issues/58","repository_url":"https://api.github.com/repos/prestodb/RPresto","labels_url":"https://api.github.com/repos/prestodb/RPresto/issues/58/labels{/name}","comments_url":"https://api.github.com/repos/prestodb/RPresto/issues/58/comments","events_url":"https://api.github.com/repos/prestodb/RPresto/issues/58/events","html_url":"https://github.com/prestodb/RPresto/pull/58","id":160835672,"number":58,"title":"Use Rcpp for 10x faster data.frame conversion","user":{"login":"saurfang","id":4317392,"avatar_url":"https://avatars.githubusercontent.com/u/4317392?v=3","gravatar_id":"","url":"https://api.github.com/users/saurfang","html_url":"https://github.com/saurfang","followers_url":"https://api.github.com/users/saurfang/followers","following_url":"https://api.github.com/users/saurfang/following{/other_user}","gists_url":"https://api.github.com/users/saurfang/gists{/gist_id}","starred_url":"https://api.github.com/users/saurfang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saurfang/subscriptions","organizations_url":"https://api.github.com/users/saurfang/orgs","repos_url":"https://api.github.com/users/saurfang/repos","events_url":"https://api.github.com/users/saurfang/events{/privacy}","received_events_url":"https://api.github.com/users/saurfang/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/prestodb/RPresto/labels/CLA%20Signed","name":"CLA Signed","color":"009900"},{"url":"https://api.github.com/repos/prestodb/RPresto/labels/GH%20Review:%20review-needed","name":"GH Review: review-needed","color":"fbca04"}],"state":"open","locked":false,"assignee":{"login":"cosinequanon","id":1226424,"avatar_url":"https://avatars.githubusercontent.com/u/1226424?v=3","gravatar_id":"","url":"https://api.github.com/users/cosinequanon","html_url":"https://github.com/cosinequanon","followers_url":"https://api.github.com/users/cosinequanon/followers","following_url":"https://api.github.com/users/cosinequanon/following{/other_user}","gists_url":"https://api.github.com/users/cosinequanon/gists{/gist_id}","starred_url":"https://api.github.com/users/cosinequanon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cosinequanon/subscriptions","organizations_url":"https://api.github.com/users/cosinequanon/orgs","repos_url":"https://api.github.com/users/cosinequanon/repos","events_url":"https://api.github.com/users/cosinequanon/events{/privacy}","received_events_url":"https://api.github.com/users/cosinequanon/received_events","type":"User","site_admin":false},"assignees":[{"login":"cosinequanon","id":1226424,"avatar_url":"https://avatars.githubusercontent.com/u/1226424?v=3","gravatar_id":"","url":"https://api.github.com/users/cosinequanon","html_url":"https://github.com/cosinequanon","followers_url":"https://api.github.com/users/cosinequanon/followers","following_url":"https://api.github.com/users/cosinequanon/following{/other_user}","gists_url":"https://api.github.com/users/cosinequanon/gists{/gist_id}","starred_url":"https://api.github.com/users/cosinequanon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cosinequanon/subscriptions","organizations_url":"https://api.github.com/users/cosinequanon/orgs","repos_url":"https://api.github.com/users/cosinequanon/repos","events_url":"https://api.github.com/users/cosinequanon/events{/privacy}","received_events_url":"https://api.github.com/users/cosinequanon/received_events","type":"User","site_admin":false}],"milestone":null,"comments":22,"created_at":"2016-06-17T08:19:25Z","updated_at":"2016-06-24T07:27:17Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/prestodb/RPresto/pulls/58","html_url":"https://github.com/prestodb/RPresto/pull/58","diff_url":"https://github.com/prestodb/RPresto/pull/58.diff","patch_url":"https://github.com/prestodb/RPresto/pull/58.patch"},"body":"Move row iterations from R to Rcpp for 10x speedup. `check_names` verify column names are consistent and each row contains exactly required columns. `transpose` convert a row major list of list to a column major list of list. It also handles null to NA conversion for inner lists and special numerical values. Because assignment is made in-place whenever possible, this is much faster than the replaced R implementation.\r\n\r\nFix #33 hopefully. I not well-versed in Rcpp so feedbacks and suggestions are really appreciated. Initial profiling result seems promising, for exact same query in #57:\r\n\r\n![image](https://cloud.githubusercontent.com/assets/4317392/16144540/81d20fbc-3428-11e6-9808-e85e71bb0e1b.png)\r\n\r\n`.extract.data` nows takes about the same time as raw JSON parsing and is 10x faster compared to original runtime and profiler credits `.tranpose` 20ms out of 110ms (note 10ms is minimal resolution). \r\nI still need to take a look at bigint and add some test cases for named/unnamed list but feel free to try this out and let me know if this fixes the performance issue or if it outputs garbage."},"comment":{"url":"https://api.github.com/repos/prestodb/RPresto/issues/comments/228276197","html_url":"https://github.com/prestodb/RPresto/pull/58#issuecomment-228276197","issue_url":"https://api.github.com/repos/prestodb/RPresto/issues/58","id":228276197,"user":{"login":"saurfang","id":4317392,"avatar_url":"https://avatars.githubusercontent.com/u/4317392?v=3","gravatar_id":"","url":"https://api.github.com/users/saurfang","html_url":"https://github.com/saurfang","followers_url":"https://api.github.com/users/saurfang/followers","following_url":"https://api.github.com/users/saurfang/following{/other_user}","gists_url":"https://api.github.com/users/saurfang/gists{/gist_id}","starred_url":"https://api.github.com/users/saurfang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saurfang/subscriptions","organizations_url":"https://api.github.com/users/saurfang/orgs","repos_url":"https://api.github.com/users/saurfang/repos","events_url":"https://api.github.com/users/saurfang/events{/privacy}","received_events_url":"https://api.github.com/users/saurfang/received_events","type":"User","site_admin":false},"created_at":"2016-06-24T07:27:17Z","updated_at":"2016-06-24T07:27:17Z","body":"Good to know. Thanks a lot for the reviews @czarandy \r\n\r\n@cosinequanon I have addressed suggestions and retested the package. I also ran some profiling and looks like `.extract.data` takes about 10-20% runtime (which is about half the time spent in `jsonlite::fromJSON`) "}}
{"actor":{"display_login":"saurfang"}}