{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21420","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21420/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21420/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21420/events","html_url":"https://github.com/apache/spark/pull/21420","id":325990324,"number":21420,"title":"[SPARK-24377][Spark Submit] make --py-files work in non pyspark application","user":{"login":"jerryshao","id":850797,"avatar_url":"https://avatars2.githubusercontent.com/u/850797?v=4","gravatar_id":"","url":"https://api.github.com/users/jerryshao","html_url":"https://github.com/jerryshao","followers_url":"https://api.github.com/users/jerryshao/followers","following_url":"https://api.github.com/users/jerryshao/following{/other_user}","gists_url":"https://api.github.com/users/jerryshao/gists{/gist_id}","starred_url":"https://api.github.com/users/jerryshao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerryshao/subscriptions","organizations_url":"https://api.github.com/users/jerryshao/orgs","repos_url":"https://api.github.com/users/jerryshao/repos","events_url":"https://api.github.com/users/jerryshao/events{/privacy}","received_events_url":"https://api.github.com/users/jerryshao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-05-24T07:07:44Z","updated_at":"2018-05-28T09:48:20Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21420","html_url":"https://github.com/apache/spark/pull/21420","diff_url":"https://github.com/apache/spark/pull/21420.diff","patch_url":"https://github.com/apache/spark/pull/21420.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nFor some Spark applications, though they're a java program, they require not only jar dependencies, but also python dependencies. One example is Livy remote SparkContext application, this application is actually an embedded REPL for Scala/Python/R, it will not only load in jar dependencies, but also python and R deps, so we should specify not only \"--jars\", but also \"--py-files\".\r\n\r\nCurrently for a Spark application, --py-files can only be worked for a pyspark application, so it will not be worked in the above case. So here propose to remove such restriction.\r\n\r\nAlso we tested that \"spark.submit.pyFiles\" only supports quite limited scenario (client mode with local deps), so here also expand the usage of \"spark.submit.pyFiles\" to be alternative of --py-files.\r\n\r\n## How was this patch tested?\r\n\r\nUT added."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/392478402","html_url":"https://github.com/apache/spark/pull/21420#issuecomment-392478402","issue_url":"https://api.github.com/repos/apache/spark/issues/21420","id":392478402,"user":{"login":"viirya","id":68855,"avatar_url":"https://avatars1.githubusercontent.com/u/68855?v=4","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"created_at":"2018-05-28T09:48:20Z","updated_at":"2018-05-28T09:48:20Z","author_association":"CONTRIBUTOR","body":"Doesn't `--files` work for adding such dependencies?"}}
{"actor":{"display_login":"viirya"}}