{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21299","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21299/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21299/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21299/events","html_url":"https://github.com/apache/spark/pull/21299","id":322292917,"number":21299,"title":"[SPARK-24250][SQL] support accessing SQLConf inside tasks","user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars3.githubusercontent.com/u/3182036?v=4","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-11T13:16:30Z","updated_at":"2018-05-11T13:18:06Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21299","html_url":"https://github.com/apache/spark/pull/21299","diff_url":"https://github.com/apache/spark/pull/21299.diff","patch_url":"https://github.com/apache/spark/pull/21299.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nPreviously in #20136 we decided to forbid tasks to access `SQLConf`, because it doesn't work and always give you the default conf value. In #21190 we fixed the check and all the places that violate it.\r\n\r\nCurrently the pattern of accessing configs at the executor side is: read the configs at the driver side, then access the variables holding the config values in the RDD closure, so that they will be serialized to the executor side. Something like\r\n```\r\nval someConf = conf.getXXX\r\nchild.execute().mapPartitions {\r\n  if (someConf == ...) ...\r\n  ...\r\n}\r\n```\r\n\r\nHowever, this pattern is hard to apply if the config needs to be propagated via a long call stack. An example is `DataType.sameType`, and see how many changes were made in #21190 .\r\n\r\nWhen it comes to code generation, it's even worse. I tried it locally and we need to change a ton of files to propagate configs to code generators.\r\n\r\nThis PR proposes to allow tasks to access `SQLConf`. The idea is, we can save all the SQL configs to job properties when an SQL execution is triggered. At executor side we rebuild the `SQLConf` from job properties.\r\n\r\nThis PR reverts #21190, please review the second commit.\r\n\r\n## How was this patch tested?\r\n\r\na new test suite"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/388360732","html_url":"https://github.com/apache/spark/pull/21299#issuecomment-388360732","issue_url":"https://api.github.com/repos/apache/spark/issues/21299","id":388360732,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-05-11T13:18:06Z","updated_at":"2018-05-11T13:18:06Z","author_association":"NONE","body":"**[Test build #90508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90508/testReport)** for PR 21299 at commit [`e2e4a52`](https://github.com/apache/spark/commit/e2e4a52e38eea0abce033baccbcb37211aa8665e)."}}
{"actor":{"display_login":"SparkQA"}}