{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla/telemetry-batch-view/issues/67","repository_url":"https://api.github.com/repos/mozilla/telemetry-batch-view","labels_url":"https://api.github.com/repos/mozilla/telemetry-batch-view/issues/67/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/telemetry-batch-view/issues/67/comments","events_url":"https://api.github.com/repos/mozilla/telemetry-batch-view/issues/67/events","html_url":"https://github.com/mozilla/telemetry-batch-view/pull/67","id":154489108,"number":67,"title":"DO NOT MERGE: Spark schema main summary to view","user":{"login":"mreid-moz","id":969479,"avatar_url":"https://avatars.githubusercontent.com/u/969479?v=3","gravatar_id":"","url":"https://api.github.com/users/mreid-moz","html_url":"https://github.com/mreid-moz","followers_url":"https://api.github.com/users/mreid-moz/followers","following_url":"https://api.github.com/users/mreid-moz/following{/other_user}","gists_url":"https://api.github.com/users/mreid-moz/gists{/gist_id}","starred_url":"https://api.github.com/users/mreid-moz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mreid-moz/subscriptions","organizations_url":"https://api.github.com/users/mreid-moz/orgs","repos_url":"https://api.github.com/users/mreid-moz/repos","events_url":"https://api.github.com/users/mreid-moz/events{/privacy}","received_events_url":"https://api.github.com/users/mreid-moz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2016-05-12T13:57:39Z","updated_at":"2016-05-19T14:30:09Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/mozilla/telemetry-batch-view/pulls/67","html_url":"https://github.com/mozilla/telemetry-batch-view/pull/67","diff_url":"https://github.com/mozilla/telemetry-batch-view/pull/67.diff","patch_url":"https://github.com/mozilla/telemetry-batch-view/pull/67.patch"},"body":"Per bug 1272334, refactor the Main Summary code to use Spark SQL types directly (rather than using Avro Schema), and switch from the \"Derived Stream\" approach to the \"Views\" approach.\r\n\r\nThis builds on the code in #62, but still has some performance problems to work out.\r\n\r\nProcessing a single day's data on a 20-node cluster (April 1, 2016 was my reference date) takes a bit more than 90 minutes with this current code.  Using the DerivedStreams approach from a few commits back takes about half that amount of time.\r\n\r\nNote that I also added a step to `Telemetry.getMessages` to balance the partition sizes, which reduced the run time from more than 3 hours down to 90 minutes or so.\r\n\r\n@vitillo @Uberi Feedback and suggestions appreciated, particularly about what I can do to identify and fix the performance difference."},"comment":{"url":"https://api.github.com/repos/mozilla/telemetry-batch-view/issues/comments/220341776","html_url":"https://github.com/mozilla/telemetry-batch-view/pull/67#issuecomment-220341776","issue_url":"https://api.github.com/repos/mozilla/telemetry-batch-view/issues/67","id":220341776,"user":{"login":"mreid-moz","id":969479,"avatar_url":"https://avatars.githubusercontent.com/u/969479?v=3","gravatar_id":"","url":"https://api.github.com/users/mreid-moz","html_url":"https://github.com/mreid-moz","followers_url":"https://api.github.com/users/mreid-moz/followers","following_url":"https://api.github.com/users/mreid-moz/following{/other_user}","gists_url":"https://api.github.com/users/mreid-moz/gists{/gist_id}","starred_url":"https://api.github.com/users/mreid-moz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mreid-moz/subscriptions","organizations_url":"https://api.github.com/users/mreid-moz/orgs","repos_url":"https://api.github.com/users/mreid-moz/repos","events_url":"https://api.github.com/users/mreid-moz/events{/privacy}","received_events_url":"https://api.github.com/users/mreid-moz/received_events","type":"User","site_admin":false},"created_at":"2016-05-19T14:30:09Z","updated_at":"2016-05-19T14:30:09Z","body":"Argh, I noticed a problem where the \"overwrite\" deletes the entire \"version\" prefix before adding data."}}