{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15998","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15998/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15998/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15998/events","html_url":"https://github.com/apache/spark/pull/15998","id":191392435,"number":15998,"title":"[SPARK-18572][SQL] Add a method `listPartitionNames` to `ExternalCatalog`","user":{"login":"mallman","id":833693,"avatar_url":"https://avatars.githubusercontent.com/u/833693?v=3","gravatar_id":"","url":"https://api.github.com/users/mallman","html_url":"https://github.com/mallman","followers_url":"https://api.github.com/users/mallman/followers","following_url":"https://api.github.com/users/mallman/following{/other_user}","gists_url":"https://api.github.com/users/mallman/gists{/gist_id}","starred_url":"https://api.github.com/users/mallman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mallman/subscriptions","organizations_url":"https://api.github.com/users/mallman/orgs","repos_url":"https://api.github.com/users/mallman/repos","events_url":"https://api.github.com/users/mallman/events{/privacy}","received_events_url":"https://api.github.com/users/mallman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":18,"created_at":"2016-11-23T22:21:57Z","updated_at":"2016-12-01T19:34:48Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15998","html_url":"https://github.com/apache/spark/pull/15998","diff_url":"https://github.com/apache/spark/pull/15998.diff","patch_url":"https://github.com/apache/spark/pull/15998.patch"},"body":"(Link to Jira issue: https://issues.apache.org/jira/browse/SPARK-18572)\r\n\r\n## What changes were proposed in this pull request?\r\n\r\nCurrently Spark answers the `SHOW PARTITIONS` command by fetching all of the table's partition metadata from the external catalog and constructing partition names therefrom. The Hive client has a `getPartitionNames` method which is many times faster for this purpose, with the performance improvement scaling with the number of partitions in a table.\r\n\r\nTo test the performance impact of this PR, I ran the `SHOW PARTITIONS` command on two Hive tables with large numbers of partitions. One table has ~17,800 partitions, and the other has ~95,000 partitions. For the purposes of this PR, I'll call the former table `table1` and the latter table `table2`. I ran 5 trials for each table with before-and-after versions of this PR. The results are as follows:\r\n\r\nSpark at bdc8153, `SHOW PARTITIONS table1`, times in seconds:\r\n7.901\r\n3.983\r\n4.018\r\n4.331\r\n4.261\r\n\r\nSpark at bdc8153, `SHOW PARTITIONS table2`\r\n(Timed out after 10 minutes with a `SocketTimeoutException`.)\r\n\r\nSpark at this PR, `SHOW PARTITIONS table1`, times in seconds:\r\n3.801\r\n0.449\r\n0.395\r\n0.348\r\n0.336\r\n\r\nSpark at this PR, `SHOW PARTITIONS table2`, times in seconds:\r\n5.184\r\n1.63\r\n1.474\r\n1.519\r\n1.41\r\n\r\nTaking the best times from each trial, we get a 12x performance improvement for a table with ~17,800 partitions and at least a 426x improvement for a table with ~95,000 partitions. More significantly, the latter command doesn't even complete with the current code in master.\r\n\r\nThis is actually a patch we've been using in-house at VideoAmp since Spark 1.1. It's made all the difference in the practical usability of our largest tables. Even with tables with about 1,000 partitions there's a performance improvement of about 2-3x.\r\n\r\n## How was this patch tested?\r\n\r\nI added a unit test to `VersionsSuite` which tests that the Hive client's `getPartitionNames` method returns the correct number of partitions."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/264270864","html_url":"https://github.com/apache/spark/pull/15998#issuecomment-264270864","issue_url":"https://api.github.com/repos/apache/spark/issues/15998","id":264270864,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-12-01T19:34:48Z","updated_at":"2016-12-01T19:34:48Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69491/\nTest FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}