{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21267","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21267/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21267/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21267/events","html_url":"https://github.com/apache/spark/pull/21267","id":321085139,"number":21267,"title":"[SPARK-21945][YARN][PYTHON] Make --py-files work with PySpark shell in Yarn client mode","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars0.githubusercontent.com/u/6477701?v=4","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2018-05-08T08:11:32Z","updated_at":"2018-05-15T01:48:05Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21267","html_url":"https://github.com/apache/spark/pull/21267","diff_url":"https://github.com/apache/spark/pull/21267.diff","patch_url":"https://github.com/apache/spark/pull/21267.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n### Problem\r\n\r\nWhen we run _PySpark shell with Yarn client mode_, specified `--py-files` are not recognised in _driver side_.\r\n\r\nHere are the steps I took to check:\r\n\r\n```bash\r\n$ cat /home/spark/tmp.py\r\ndef testtest():\r\n    return 1\r\n```\r\n\r\n```bash\r\n$ ./bin/pyspark --master yarn --deploy-mode client --py-files /home/spark/tmp.py\r\n```\r\n\r\n```python\r\n>>> def test():\r\n...     import tmp\r\n...     return tmp.testtest()\r\n...\r\n>>> spark.range(1).rdd.map(lambda _: test()).collect()  # executor side\r\n[1]\r\n>>> test()  # driver side\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"<stdin>\", line 2, in test\r\nImportError: No module named tmp\r\n```\r\n\r\n### How did it happen?\r\n\r\nUnlike Yarn cluster and client mode with Spark submit, when Yarn client mode with PySpark shell specifically,\r\n\r\n1. It first runs Python shell via:\r\n\r\nhttps://github.com/apache/spark/blob/3cb82047f2f51af553df09b9323796af507d36f8/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L158 as pointed out by @tgravescs in the JIRA.\r\n\r\n2. this triggers shell.py and submit another application to launch a py4j gateway:\r\n\r\nhttps://github.com/apache/spark/blob/209b9361ac8a4410ff797cff1115e1888e2f7e66/python/pyspark/java_gateway.py#L45-L60\r\n\r\n3. it runs a Py4J gateway:\r\n\r\nhttps://github.com/apache/spark/blob/3cb82047f2f51af553df09b9323796af507d36f8/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L425\r\n\r\n4. it copies (or downloads) --py-files  into local temp directory:\r\n\r\nhttps://github.com/apache/spark/blob/3cb82047f2f51af553df09b9323796af507d36f8/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L365-L376\r\n\r\nand then these files are set up to `spark.submit.pyFiles`\r\n\r\n5. Py4J JVM is launched and then the Python paths are set via:\r\n\r\nhttps://github.com/apache/spark/blob/7013eea11cb32b1e0038dc751c485da5c94a484b/python/pyspark/context.py#L209-L216\r\n\r\nHowever, these are not actually set because those files were copied into a tmp directory in 4. whereas this code path looks for `SparkFiles.getRootDirectory` where the files are stored only when `SparkContext.addFile()` is called.\r\n\r\nIn other cluster mode, `spark.files` are set via:\r\n\r\nhttps://github.com/apache/spark/blob/3cb82047f2f51af553df09b9323796af507d36f8/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L554-L555\r\n\r\nand those files are explicitly added via:\r\n\r\nhttps://github.com/apache/spark/blob/ecb8b383af1cf1b67f3111c148229e00c9c17c40/core/src/main/scala/org/apache/spark/SparkContext.scala#L395\r\n\r\nSo we are fine in other modes.\r\n\r\nIn case of Yarn client and cluster with _submit_, these are manually being handled. In particular https://github.com/apache/spark/pull/6360 added most of the logics. In this case, the Python path looks manually set via, for example, `deploy.PythonRunner`. We don't use `spark.files` here. \r\n\r\n### How does the PR fix the problem?\r\n\r\nI tried to make an isolated approach as possible as I can: simply copy py file or zip files into `SparkFiles.getRootDirectory()` in driver side if not existing. Another possible way is to set `spark.files` but it does unnecessary stuff together and sounds a bit invasive.\r\n\r\n**Before**\r\n\r\n```python\r\n>>> def test():\r\n...     import tmp\r\n...     return tmp.testtest()\r\n...\r\n>>> spark.range(1).rdd.map(lambda _: test()).collect()\r\n[1]\r\n>>> test()\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"<stdin>\", line 2, in test\r\nImportError: No module named tmp\r\n```\r\n\r\n**After**\r\n\r\n```python\r\n>>> def test():\r\n...     import tmp\r\n...     return tmp.testtest()\r\n...\r\n>>> spark.range(1).rdd.map(lambda _: test()).collect()\r\n[1]\r\n>>> test()\r\n1\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nI manually tested in standalone and yarn cluster with PySpark shell. .zip and .py files were also tested with the similar steps above. It's difficult to add a test."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/389016471","html_url":"https://github.com/apache/spark/pull/21267#issuecomment-389016471","issue_url":"https://api.github.com/repos/apache/spark/issues/21267","id":389016471,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-05-15T01:48:05Z","updated_at":"2018-05-15T01:48:05Z","author_association":"NONE","body":"**[Test build #90616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90616/testReport)** for PR 21267 at commit [`ef3555e`](https://github.com/apache/spark/commit/ef3555e389ea36159e9a1dfd076e9f6afbaf3f35)."}}
{"actor":{"display_login":"SparkQA"}}