{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13491","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13491/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13491/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13491/events","html_url":"https://github.com/apache/spark/pull/13491","id":158295840,"number":13491,"title":"[SPARK-15748][SQL] Replace inefficient foldLeft() call with flatMap() in PartitionStatistics","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2016-06-03T05:48:35Z","updated_at":"2016-06-03T21:53:21Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13491","html_url":"https://github.com/apache/spark/pull/13491","diff_url":"https://github.com/apache/spark/pull/13491.diff","patch_url":"https://github.com/apache/spark/pull/13491.patch"},"body":"`PartitionStatistics` uses `foldLeft` and list concatenation (`++`) to flatten an iterator of lists, but this is extremely inefficient compared to simply doing `flatMap`/`flatten` because it performs many unnecessary object allocations. Simply replacing this `foldLeft` by a `flatMap` results in decent performance gains when constructing PartitionStatistics instances for tables with many columns.\r\n\r\nThis patch fixes this and also makes two similar changes in MLlib and streaming to try to fix all known occurrences of this pattern."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/223703825","html_url":"https://github.com/apache/spark/pull/13491#issuecomment-223703825","issue_url":"https://api.github.com/repos/apache/spark/issues/13491","id":223703825,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-06-03T21:53:20Z","updated_at":"2016-06-03T21:53:20Z","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}