{"action":"created","issue":{"url":"https://api.github.com/repos/rstudio/sparklyr/issues/941","repository_url":"https://api.github.com/repos/rstudio/sparklyr","labels_url":"https://api.github.com/repos/rstudio/sparklyr/issues/941/labels{/name}","comments_url":"https://api.github.com/repos/rstudio/sparklyr/issues/941/comments","events_url":"https://api.github.com/repos/rstudio/sparklyr/issues/941/events","html_url":"https://github.com/rstudio/sparklyr/issues/941","id":250778682,"node_id":"MDU6SXNzdWUyNTA3Nzg2ODI=","number":941,"title":"Improve Serialization","user":{"login":"javierluraschi","id":3478847,"node_id":"MDQ6VXNlcjM0Nzg4NDc=","avatar_url":"https://avatars1.githubusercontent.com/u/3478847?v=4","gravatar_id":"","url":"https://api.github.com/users/javierluraschi","html_url":"https://github.com/javierluraschi","followers_url":"https://api.github.com/users/javierluraschi/followers","following_url":"https://api.github.com/users/javierluraschi/following{/other_user}","gists_url":"https://api.github.com/users/javierluraschi/gists{/gist_id}","starred_url":"https://api.github.com/users/javierluraschi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/javierluraschi/subscriptions","organizations_url":"https://api.github.com/users/javierluraschi/orgs","repos_url":"https://api.github.com/users/javierluraschi/repos","events_url":"https://api.github.com/users/javierluraschi/events{/privacy}","received_events_url":"https://api.github.com/users/javierluraschi/received_events","type":"User","site_admin":false},"labels":[{"id":601825355,"node_id":"MDU6TGFiZWw2MDE4MjUzNTU=","url":"https://api.github.com/repos/rstudio/sparklyr/labels/featurerequest","name":"featurerequest","color":"006b75","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":20,"created_at":"2017-08-16T22:30:08Z","updated_at":"2018-06-26T13:53:38Z","closed_at":"2018-06-26T13:53:38Z","author_association":"MEMBER","body":"There are two areas worth considering here:\r\n\r\n1. Improve `collect()`serialization. This is already implemented as columnar-based collect; however, I believe only for numeric and logical data types.\r\n\r\n2. Implement `copy_to()` through columnar `invoke()`.\r\n\r\nAssuming (2) is on a par with current implementations of `copy_to()`, we can stop here; otherwise, we would have to explore additional serialization improvements.\r\n\r\nRelated issues:\r\n- https://github.com/rstudio/sparklyr/issues/202 and https://github.com/rstudio/sparklyr/issues/187: Should be fixed with this work item since `Date` would serialize properly into Spark.\r\n- https://github.com/rstudio/sparklyr/issues/151, https://github.com/rstudio/sparklyr/issues/163, https://github.com/rstudio/sparklyr/issues/506, https://github.com/rstudio/sparklyr/issues/528 https://github.com/rstudio/sparklyr/issues/680, https://github.com/rstudio/sparklyr/issues/1143: Issues with `\"\"` and `NA` should be addressed by a common serialization mechanism. Notice also that some `NA` in `flights` are being copied as `NaN`.\r\n- https://github.com/rstudio/sparklyr/issues/304 not clear what the issue in `robotics.csv` was nor we have a repro, but this change should help fix that.\r\n- https://github.com/rstudio/sparklyr/issues/307 titanic data under `copy_to`, improving fidelity here would solve this issue as well.\r\n- https://github.com/rstudio/sparklyr/issues/515 Revisit special types: `NaN`, `NA`, `NULL` and `\"\"`.\r\n- https://github.com/rstudio/sparklyr/issues/1020 `\\n` in `copy_to`\r\n- https://github.com/rstudio/sparklyr/issues/1013 Serialize correctly `DATE`, `DATETIME`, and `TIMESTAMP` while coming from JDBC. Probably no need to do anything JDBC specific, but worth testing\r\n- https://github.com/rstudio/sparklyr/issues/1030 handle vector columns in `copy_to()`\r\n- https://github.com/rstudio/sparklyr/issues/1034 column based copy_to could be reused to speed up `spark_apply()`\r\n- [Collect NA in logical columns](https://stackoverflow.com/questions/47539670/convert-a-string-to-logical-in-r-with-sparklyr/47559040)."},"comment":{"url":"https://api.github.com/repos/rstudio/sparklyr/issues/comments/400317062","html_url":"https://github.com/rstudio/sparklyr/issues/941#issuecomment-400317062","issue_url":"https://api.github.com/repos/rstudio/sparklyr/issues/941","id":400317062,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMDMxNzA2Mg==","user":{"login":"javierluraschi","id":3478847,"node_id":"MDQ6VXNlcjM0Nzg4NDc=","avatar_url":"https://avatars1.githubusercontent.com/u/3478847?v=4","gravatar_id":"","url":"https://api.github.com/users/javierluraschi","html_url":"https://github.com/javierluraschi","followers_url":"https://api.github.com/users/javierluraschi/followers","following_url":"https://api.github.com/users/javierluraschi/following{/other_user}","gists_url":"https://api.github.com/users/javierluraschi/gists{/gist_id}","starred_url":"https://api.github.com/users/javierluraschi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/javierluraschi/subscriptions","organizations_url":"https://api.github.com/users/javierluraschi/orgs","repos_url":"https://api.github.com/users/javierluraschi/repos","events_url":"https://api.github.com/users/javierluraschi/events{/privacy}","received_events_url":"https://api.github.com/users/javierluraschi/received_events","type":"User","site_admin":false},"created_at":"2018-06-26T13:53:38Z","updated_at":"2018-06-26T13:53:38Z","author_association":"MEMBER","body":"We are planning to improve serialization by using Apache Arrow, this work should address many conversion issues by providing a common serialization format between R and Scale that we don't have to maintain in `sparklyr`. Arrow work is being tracked with https://github.com/rstudio/sparklyr/issues/1457."}}
{"actor":{"display_login":"javierluraschi"}}