{"action":"created","issue":{"url":"https://api.github.com/repos/rajadain/mmw-geoprocessing/issues/1","repository_url":"https://api.github.com/repos/rajadain/mmw-geoprocessing","labels_url":"https://api.github.com/repos/rajadain/mmw-geoprocessing/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/rajadain/mmw-geoprocessing/issues/1/comments","events_url":"https://api.github.com/repos/rajadain/mmw-geoprocessing/issues/1/events","html_url":"https://github.com/rajadain/mmw-geoprocessing/pull/1","id":152459046,"number":1,"title":"Serialize Less Data","user":{"login":"jamesmcclain","id":11281373,"avatar_url":"https://avatars.githubusercontent.com/u/11281373?v=3","gravatar_id":"","url":"https://api.github.com/users/jamesmcclain","html_url":"https://github.com/jamesmcclain","followers_url":"https://api.github.com/users/jamesmcclain/followers","following_url":"https://api.github.com/users/jamesmcclain/following{/other_user}","gists_url":"https://api.github.com/users/jamesmcclain/gists{/gist_id}","starred_url":"https://api.github.com/users/jamesmcclain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamesmcclain/subscriptions","organizations_url":"https://api.github.com/users/jamesmcclain/orgs","repos_url":"https://api.github.com/users/jamesmcclain/repos","events_url":"https://api.github.com/users/jamesmcclain/events{/privacy}","received_events_url":"https://api.github.com/users/jamesmcclain/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":18,"created_at":"2016-05-01T22:29:18Z","updated_at":"2016-05-03T00:44:34Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/rajadain/mmw-geoprocessing/pulls/1","html_url":"https://github.com/rajadain/mmw-geoprocessing/pull/1","diff_url":"https://github.com/rajadain/mmw-geoprocessing/pull/1.diff","patch_url":"https://github.com/rajadain/mmw-geoprocessing/pull/1.patch"},"body":"This should be have been noticed earlier, but it appears that a lot of the performance issues were/are actually due to serializing too much data.\r\n\r\nIn this pull request, changes are made to prevent the entire set of (multi-)lines from being serialized and deserialized for every Spark task.  Instead, only those (multi-)lines which are relevant for a specific key are sent along to the task that rasterizes the lines over the associated tiles.   (I am not sure if \"task\" is the correct Spark-specific nomenclature, but I hope that my meaning is understood.)\r\n\r\n~~There is still more that can be done here: the r-tree is not longer required.  Instead, it should be possible to sort the list of lines into appropriate sub-lists directly rather than constructing and querying an r-tree.~~\r\n\r\nAfter a few warm-up runs, this code produces the following result on my work machine\r\n```\r\n{\r\n  \"result\": {\r\n    \"List(42)\": 492,\r\n    \"List(22)\": 15417,\r\n    \"List(43)\": 1934,\r\n    \"List(71)\": 412,\r\n    \"List(41)\": 75479,\r\n    \"List(21)\": 45149,\r\n    \"List(0)\": 75,\r\n    \"List(24)\": 2529,\r\n    \"List(31)\": 173,\r\n    \"List(90)\": 27144,\r\n    \"List(52)\": 11009,\r\n    \"List(11)\": 3551,\r\n    \"List(23)\": 8442,\r\n    \"List(82)\": 12579,\r\n    \"List(81)\": 18622,\r\n    \"List(95)\": 709\r\n  }\r\n}curl --data-binary @/tmp/MapshedJob_DRB_VeryLarge.json   0.01s user 0.03s system 0% cpu 4.974 total\r\n```\r\nwhereas it could not complete in 10 seconds before."},"comment":{"url":"https://api.github.com/repos/rajadain/mmw-geoprocessing/issues/comments/216405996","html_url":"https://github.com/rajadain/mmw-geoprocessing/pull/1#issuecomment-216405996","issue_url":"https://api.github.com/repos/rajadain/mmw-geoprocessing/issues/1","id":216405996,"user":{"login":"jamesmcclain","id":11281373,"avatar_url":"https://avatars.githubusercontent.com/u/11281373?v=3","gravatar_id":"","url":"https://api.github.com/users/jamesmcclain","html_url":"https://github.com/jamesmcclain","followers_url":"https://api.github.com/users/jamesmcclain/followers","following_url":"https://api.github.com/users/jamesmcclain/following{/other_user}","gists_url":"https://api.github.com/users/jamesmcclain/gists{/gist_id}","starred_url":"https://api.github.com/users/jamesmcclain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamesmcclain/subscriptions","organizations_url":"https://api.github.com/users/jamesmcclain/orgs","repos_url":"https://api.github.com/users/jamesmcclain/repos","events_url":"https://api.github.com/users/jamesmcclain/events{/privacy}","received_events_url":"https://api.github.com/users/jamesmcclain/received_events","type":"User","site_admin":false},"created_at":"2016-05-03T00:44:33Z","updated_at":"2016-05-03T00:44:33Z","body":"It is your choice, but I would be for keeping both the sequential and parallel version and perhaps choosing between them in the frontend.  As implied by the charts above, the particular case in which I see the parallel version being very useful is when there is a large number of tiles but a fairly sparse collection of streams (e.g. using the NHD dataset over a large AoI).\r\n\r\nIf GitHub had a \"drop the mic\" icon I would now use it."}}