{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14117","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14117/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14117/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14117/events","html_url":"https://github.com/apache/spark/pull/14117","id":164667683,"number":14117,"title":"[SPARK-16461][SQL] Support partition batch pruning with `<=>` predicate in InMemoryTableScanExec","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-07-09T13:45:12Z","updated_at":"2016-07-09T21:20:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14117","html_url":"https://github.com/apache/spark/pull/14117","diff_url":"https://github.com/apache/spark/pull/14117.diff","patch_url":"https://github.com/apache/spark/pull/14117.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIt seems `EqualNullSafe` filter was missed for batch pruneing partitions in cached tables.\r\n\r\nIt seems supporting this improves the performance roughly 5 times faster.\r\n\r\nRunning the codes below:\r\n\r\n```scala\r\ntest(\"Null-safe equal comparison\") {\r\n  val N = 20000000\r\n  val df = spark.range(N).repartition(20)\r\n  val benchmark = new Benchmark(\"Null-safe equal comparison\", N)\r\n  df.createOrReplaceTempView(\"t\")\r\n  spark.catalog.cacheTable(\"t\")\r\n  sql(\"select id from t where id <=> 1\").collect()\r\n\r\n  benchmark.addCase(\"Null-safe equal comparison\", 10) { _ =>\r\n    sql(\"select id from t where id <=> 1\").collect()\r\n  }\r\n  benchmark.run()\r\n}\r\n```\r\n\r\n\r\nproduces the results below:\r\n\r\n**Before:**\r\n\r\n```\r\nRunning benchmark: Null-safe equal comparison\r\n  Running case: Null-safe equal comparison\r\n  Stopped after 10 iterations, 2098 ms\r\n\r\nJava HotSpot(TM) 64-Bit Server VM 1.8.0_45-b14 on Mac OS X 10.11.5\r\nIntel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz\r\n\r\nNull-safe equal comparison:              Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nNull-safe equal comparison                     204 /  210         98.1          10.2       1.0X\r\n```\r\n\r\n**After:**\r\n\r\n```\r\nRunning benchmark: Null-safe equal comparison\r\n  Running case: Null-safe equal comparison\r\n  Stopped after 10 iterations, 478 ms\r\n\r\nJava HotSpot(TM) 64-Bit Server VM 1.8.0_45-b14 on Mac OS X 10.11.5\r\nIntel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz\r\n\r\nNull-safe equal comparison:              Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nNull-safe equal comparison                      42 /   48        474.1           2.1       1.0X\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nUnit tests in `PartitionBatchPruningSuite`."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/231556490","html_url":"https://github.com/apache/spark/pull/14117#issuecomment-231556490","issue_url":"https://api.github.com/repos/apache/spark/issues/14117","id":231556490,"user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"created_at":"2016-07-09T21:20:51Z","updated_at":"2016-07-09T21:20:51Z","body":"Thank you! Could you take a look @davies and @liancheng ?"}}
{"actor":{"display_login":"HyukjinKwon"}}