{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22313","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22313/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22313/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22313/events","html_url":"https://github.com/apache/spark/pull/22313","id":356237436,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEyNTY3MjY1","number":22313,"title":"[SPARK-25306][SQL] Use cache to speed up `createFilter` in ORC","user":{"login":"dongjoon-hyun","id":9700541,"node_id":"MDQ6VXNlcjk3MDA1NDE=","avatar_url":"https://avatars2.githubusercontent.com/u/9700541?v=4","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-09-01T22:35:12Z","updated_at":"2018-09-02T02:32:45Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22313","html_url":"https://github.com/apache/spark/pull/22313","diff_url":"https://github.com/apache/spark/pull/22313.diff","patch_url":"https://github.com/apache/spark/pull/22313.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIn both ORC data sources, `createFilter` function has exponential time complexity due to lack of memoization like the following. This PR aims to improve it by using cache with new configuration `spark.sql.orc.cache.sarg.timeout` (default: `20s` and `0s` means the previous behavior.)\r\n\r\n**REPRODUCE**\r\n```scala\r\n// Create and read 1 row table with 1000 columns\r\nsql(\"set spark.sql.orc.filterPushdown=true\")\r\nval selectExpr = (1 to 1000).map(i => s\"id c$i\")\r\nspark.range(1).selectExpr(selectExpr: _*).write.mode(\"overwrite\").orc(\"/tmp/orc\")\r\nprint(s\"With 0 filters, \")\r\nspark.time(spark.read.orc(\"/tmp/orc\").count)\r\n\r\n// Increase the number of filters\r\n(20 to 30).foreach { width =>\r\n  val whereExpr = (1 to width).map(i => s\"c$i is not null\").mkString(\" and \")\r\n  print(s\"With $width filters, \")\r\n  spark.time(spark.read.orc(\"/tmp/orc\").where(whereExpr).count)\r\n}\r\n```\r\n\r\n**RESULT**\r\n```scala\r\nWith 0 filters, Time taken: 653 ms                                              \r\nWith 20 filters, Time taken: 962 ms\r\nWith 21 filters, Time taken: 1282 ms\r\nWith 22 filters, Time taken: 1982 ms\r\nWith 23 filters, Time taken: 3855 ms\r\nWith 24 filters, Time taken: 6719 ms\r\nWith 25 filters, Time taken: 12669 ms\r\nWith 26 filters, Time taken: 25032 ms\r\nWith 27 filters, Time taken: 49585 ms\r\nWith 28 filters, Time taken: 98980 ms    // over 1 min 38 seconds\r\nWith 29 filters, Time taken: 198368 ms   // over 3 mins\r\nWith 30 filters, Time taken: 393744 ms   // over 6 mins\r\n```\r\n\r\n**AFTER THIS PR**\r\n```scala\r\nWith 0 filters, Time taken: 644 ms                                              \r\nWith 20 filters, Time taken: 638 ms\r\nWith 21 filters, Time taken: 360 ms\r\nWith 22 filters, Time taken: 590 ms\r\nWith 23 filters, Time taken: 318 ms\r\nWith 24 filters, Time taken: 315 ms\r\nWith 25 filters, Time taken: 381 ms\r\nWith 26 filters, Time taken: 304 ms\r\nWith 27 filters, Time taken: 294 ms\r\nWith 28 filters, Time taken: 319 ms\r\nWith 29 filters, Time taken: 288 ms\r\nWith 30 filters, Time taken: 285 ms\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nPass the Jenkins with newly added test cases."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/417900499","html_url":"https://github.com/apache/spark/pull/22313#issuecomment-417900499","issue_url":"https://api.github.com/repos/apache/spark/issues/22313","id":417900499,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNzkwMDQ5OQ==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-09-02T02:32:45Z","updated_at":"2018-09-02T02:32:45Z","author_association":"NONE","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}