{"action":"created","issue":{"url":"https://api.github.com/repos/astrolabsoftware/spark3D/issues/81","repository_url":"https://api.github.com/repos/astrolabsoftware/spark3D","labels_url":"https://api.github.com/repos/astrolabsoftware/spark3D/issues/81/labels{/name}","comments_url":"https://api.github.com/repos/astrolabsoftware/spark3D/issues/81/comments","events_url":"https://api.github.com/repos/astrolabsoftware/spark3D/issues/81/events","html_url":"https://github.com/astrolabsoftware/spark3D/pull/81","id":344744052,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA0MDY0NDc1","number":81,"title":"Cache the rawRDD by default to speed-up re-partitioning.","user":{"login":"JulienPeloton","id":20426972,"node_id":"MDQ6VXNlcjIwNDI2OTcy","avatar_url":"https://avatars3.githubusercontent.com/u/20426972?v=4","gravatar_id":"","url":"https://api.github.com/users/JulienPeloton","html_url":"https://github.com/JulienPeloton","followers_url":"https://api.github.com/users/JulienPeloton/followers","following_url":"https://api.github.com/users/JulienPeloton/following{/other_user}","gists_url":"https://api.github.com/users/JulienPeloton/gists{/gist_id}","starred_url":"https://api.github.com/users/JulienPeloton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JulienPeloton/subscriptions","organizations_url":"https://api.github.com/users/JulienPeloton/orgs","repos_url":"https://api.github.com/users/JulienPeloton/repos","events_url":"https://api.github.com/users/JulienPeloton/events{/privacy}","received_events_url":"https://api.github.com/users/JulienPeloton/received_events","type":"User","site_admin":false},"labels":[{"id":934740489,"node_id":"MDU6TGFiZWw5MzQ3NDA0ODk=","url":"https://api.github.com/repos/astrolabsoftware/spark3D/labels/Partitioning","name":"Partitioning","color":"f9d0c4","default":false},{"id":997566449,"node_id":"MDU6TGFiZWw5OTc1NjY0NDk=","url":"https://api.github.com/repos/astrolabsoftware/spark3D/labels/performance","name":"performance","color":"2493f4","default":false}],"state":"open","locked":false,"assignee":{"login":"JulienPeloton","id":20426972,"node_id":"MDQ6VXNlcjIwNDI2OTcy","avatar_url":"https://avatars3.githubusercontent.com/u/20426972?v=4","gravatar_id":"","url":"https://api.github.com/users/JulienPeloton","html_url":"https://github.com/JulienPeloton","followers_url":"https://api.github.com/users/JulienPeloton/followers","following_url":"https://api.github.com/users/JulienPeloton/following{/other_user}","gists_url":"https://api.github.com/users/JulienPeloton/gists{/gist_id}","starred_url":"https://api.github.com/users/JulienPeloton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JulienPeloton/subscriptions","organizations_url":"https://api.github.com/users/JulienPeloton/orgs","repos_url":"https://api.github.com/users/JulienPeloton/repos","events_url":"https://api.github.com/users/JulienPeloton/events{/privacy}","received_events_url":"https://api.github.com/users/JulienPeloton/received_events","type":"User","site_admin":false},"assignees":[{"login":"JulienPeloton","id":20426972,"node_id":"MDQ6VXNlcjIwNDI2OTcy","avatar_url":"https://avatars3.githubusercontent.com/u/20426972?v=4","gravatar_id":"","url":"https://api.github.com/users/JulienPeloton","html_url":"https://github.com/JulienPeloton","followers_url":"https://api.github.com/users/JulienPeloton/followers","following_url":"https://api.github.com/users/JulienPeloton/following{/other_user}","gists_url":"https://api.github.com/users/JulienPeloton/gists{/gist_id}","starred_url":"https://api.github.com/users/JulienPeloton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JulienPeloton/subscriptions","organizations_url":"https://api.github.com/users/JulienPeloton/orgs","repos_url":"https://api.github.com/users/JulienPeloton/repos","events_url":"https://api.github.com/users/JulienPeloton/events{/privacy}","received_events_url":"https://api.github.com/users/JulienPeloton/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-07-26T08:26:56Z","updated_at":"2018-07-26T08:33:41Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/astrolabsoftware/spark3D/pulls/81","html_url":"https://github.com/astrolabsoftware/spark3D/pull/81","diff_url":"https://github.com/astrolabsoftware/spark3D/pull/81.diff","patch_url":"https://github.com/astrolabsoftware/spark3D/pull/81.patch"},"body":"When re-partitioning, we typically needs to operate on the rawRDD (e.g. gather statistics) several times, so I've added persistence on the rawRDD (default is StorageLevel.MEMORY_ONLY) to reduce the execution time. Here is a summary on re-partitioning 10 million Point3D:\r\n\r\n| | no repartition | Onion | Octree |\r\n|:--:|:--:|:--:|:--:|\r\n| previous time (s) | 15 | 70 | 96 |\r\n| new time (s) | 15 | 50 | 57|\r\n\r\nOctree re-partitioning benefits a lot from this as we have to perform `count`, `takeSample` and `aggregate` prior to the re-partitioning. \r\n"},"comment":{"url":"https://api.github.com/repos/astrolabsoftware/spark3D/issues/comments/408021042","html_url":"https://github.com/astrolabsoftware/spark3D/pull/81#issuecomment-408021042","issue_url":"https://api.github.com/repos/astrolabsoftware/spark3D/issues/81","id":408021042,"node_id":"MDEyOklzc3VlQ29tbWVudDQwODAyMTA0Mg==","user":{"login":"codecov-io","id":8655789,"node_id":"MDQ6VXNlcjg2NTU3ODk=","avatar_url":"https://avatars0.githubusercontent.com/u/8655789?v=4","gravatar_id":"","url":"https://api.github.com/users/codecov-io","html_url":"https://github.com/codecov-io","followers_url":"https://api.github.com/users/codecov-io/followers","following_url":"https://api.github.com/users/codecov-io/following{/other_user}","gists_url":"https://api.github.com/users/codecov-io/gists{/gist_id}","starred_url":"https://api.github.com/users/codecov-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/codecov-io/subscriptions","organizations_url":"https://api.github.com/users/codecov-io/orgs","repos_url":"https://api.github.com/users/codecov-io/repos","events_url":"https://api.github.com/users/codecov-io/events{/privacy}","received_events_url":"https://api.github.com/users/codecov-io/received_events","type":"User","site_admin":false},"created_at":"2018-07-26T08:33:41Z","updated_at":"2018-07-26T08:33:41Z","author_association":"NONE","body":"# [Codecov](https://codecov.io/gh/astrolabsoftware/spark3D/pull/81?src=pr&el=h1) Report\n> Merging [#81](https://codecov.io/gh/astrolabsoftware/spark3D/pull/81?src=pr&el=desc) into [master](https://codecov.io/gh/astrolabsoftware/spark3D/commit/ecc343c5256b1e0b61ed7869eec5efb26104491c?src=pr&el=desc) will **increase** coverage by `0.01%`.\n> The diff coverage is `80%`.\n\n[![Impacted file tree graph](https://codecov.io/gh/astrolabsoftware/spark3D/pull/81/graphs/tree.svg?token=CHwrMSMSkX&src=pr&width=650&height=150)](https://codecov.io/gh/astrolabsoftware/spark3D/pull/81?src=pr&el=tree)\n\n```diff\n@@           Coverage Diff           @@\n##           master   #81      +/-   ##\n=======================================\n+ Coverage   96.99%   97%   +0.01%     \n=======================================\n  Files          23    23              \n  Lines         831   834       +3     \n  Branches      225   223       -2     \n=======================================\n+ Hits          806   809       +3     \n  Misses         25    25\n```\n\n\n| [Impacted Files](https://codecov.io/gh/astrolabsoftware/spark3D/pull/81?src=pr&el=tree) | Coverage Δ | |\n|---|---|---|\n| [...in/scala/com/spark3d/spatial3DRDD/Shape3DRDD.scala](https://codecov.io/gh/astrolabsoftware/spark3D/pull/81/diff?src=pr&el=tree#diff-c3JjL21haW4vc2NhbGEvY29tL3NwYXJrM2Qvc3BhdGlhbDNEUkREL1NoYXBlM0RSREQuc2NhbGE=) | `95.23% <100%> (ø)` | :arrow_up: |\n| [...ain/scala/com/spark3d/spatial3DRDD/SphereRDD.scala](https://codecov.io/gh/astrolabsoftware/spark3D/pull/81/diff?src=pr&el=tree#diff-c3JjL21haW4vc2NhbGEvY29tL3NwYXJrM2Qvc3BhdGlhbDNEUkREL1NwaGVyZVJERC5zY2FsYQ==) | `80% <66.66%> (+5%)` | :arrow_up: |\n| [...in/scala/com/spark3d/spatial3DRDD/Point3DRDD.scala](https://codecov.io/gh/astrolabsoftware/spark3D/pull/81/diff?src=pr&el=tree#diff-c3JjL21haW4vc2NhbGEvY29tL3NwYXJrM2Qvc3BhdGlhbDNEUkREL1BvaW50M0RSREQuc2NhbGE=) | `83.33% <80%> (+8.33%)` | :arrow_up: |\n\n------\n\n[Continue to review full report at Codecov](https://codecov.io/gh/astrolabsoftware/spark3D/pull/81?src=pr&el=continue).\n> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)\n> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`\n> Powered by [Codecov](https://codecov.io/gh/astrolabsoftware/spark3D/pull/81?src=pr&el=footer). Last update [ecc343c...885c146](https://codecov.io/gh/astrolabsoftware/spark3D/pull/81?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).\n"}}
{"actor":{"display_login":"codecov-io"}}