{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13887","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13887/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13887/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13887/events","html_url":"https://github.com/apache/spark/pull/13887","id":162100735,"number":13887,"title":"[SPARK-16186][SQL] Support partition batch pruning with `IN` predicate in InMemoryTableScanExec","user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars.githubusercontent.com/u/9700541?v=3","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-06-24T08:35:03Z","updated_at":"2016-06-24T11:08:38Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13887","html_url":"https://github.com/apache/spark/pull/13887","diff_url":"https://github.com/apache/spark/pull/13887.diff","patch_url":"https://github.com/apache/spark/pull/13887.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nOne of the most frequent usage patterns for Spark SQL is using **cached tables**. This PR improves `InMemoryTableScanExec` to handle `IN` predicate efficiently by pruning partition batches. Of course, the performance improvement varies over the queries and the datasets. But, for the following simple query, the query duration in Spark UI goes from 9 seconds to 50~90ms. It's about over 100 times faster.\r\n```scala\r\n$ bin/spark-shell --driver-memory 6G\r\nscala> val df = spark.range(2000000000)\r\nscala> df.createOrReplaceTempView(\"t\")\r\nscala> spark.catalog.cacheTable(\"t\")\r\nscala> sql(\"select id from t where id = 1\").collect()    // About 2 mins\r\nscala> sql(\"select id from t where id = 1\").collect()    // less than 90ms\r\nscala> sql(\"select id from t where id in (1,2,3)\").collect()  // 9 seconds\r\nscala> spark.conf.set(\"spark.sql.inMemoryColumnarStorage.partitionPruningMaxInSize\", 10)  // Enable. (Just to show this examples, currently the default value is 10.)\r\nscala> sql(\"select id from t where id in (1,2,3)\").collect() // less than 90ms\r\nscala> spark.conf.set(\"spark.sql.inMemoryColumnarStorage.partitionPruningMaxInSize\", 0)  // Disable\r\nscala> sql(\"select id from t where id in (1,2,3)\").collect() // 9 seconds\r\n```\r\n\r\nThis PR has impacts over 35 queries of TPC-DS if the tables are cached.\r\n\r\n## How was this patch tested?\r\n\r\nPass the Jenkins tests (including new testcases)."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/228318764","html_url":"https://github.com/apache/spark/pull/13887#issuecomment-228318764","issue_url":"https://api.github.com/repos/apache/spark/issues/13887","id":228318764,"user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars.githubusercontent.com/u/9700541?v=3","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"created_at":"2016-06-24T11:08:38Z","updated_at":"2016-06-24T11:08:38Z","body":"Thank you, @cloud-fan !"}}
{"actor":{"display_login":"dongjoon-hyun"}}