{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12678","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12678/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12678/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12678/events","html_url":"https://github.com/apache/spark/pull/12678","id":151018748,"number":12678,"title":"[SPARK-14908] [YARN] Provide support HDFS-located resources for spark…","user":{"login":"mikhaildubkov","id":4924409,"avatar_url":"https://avatars.githubusercontent.com/u/4924409?v=3","gravatar_id":"","url":"https://api.github.com/users/mikhaildubkov","html_url":"https://github.com/mikhaildubkov","followers_url":"https://api.github.com/users/mikhaildubkov/followers","following_url":"https://api.github.com/users/mikhaildubkov/following{/other_user}","gists_url":"https://api.github.com/users/mikhaildubkov/gists{/gist_id}","starred_url":"https://api.github.com/users/mikhaildubkov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikhaildubkov/subscriptions","organizations_url":"https://api.github.com/users/mikhaildubkov/orgs","repos_url":"https://api.github.com/users/mikhaildubkov/repos","events_url":"https://api.github.com/users/mikhaildubkov/events{/privacy}","received_events_url":"https://api.github.com/users/mikhaildubkov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-04-26T01:25:47Z","updated_at":"2016-04-27T22:09:01Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12678","html_url":"https://github.com/apache/spark/pull/12678","diff_url":"https://github.com/apache/spark/pull/12678.diff","patch_url":"https://github.com/apache/spark/pull/12678.patch"},"body":"The main goal behind these changes are provide support to use HDFS resources for \"spark.executor.extraClassPath\", when Hadoop/YARN deployments used.\r\n This can be helpful when you want to use custom SparkSerializer implementation (our project case).\r\n\r\nHow it works with these changes:\r\n1. Value of \"spark.executor.extraClassPath\" splits by comma\r\n2. Iterate over all paths and filter those which started with \"hdfs;//\"\r\n3. Generate link for each path and add LocalResource to executor launch context local resources\r\n4. Add generated links to executor CLASSPATH\r\n5. NodeManager loads the specified local resources to application cache\r\n\r\nAfter that, you do not need deploy extra resources to each Hadoop node manually, it will be automatically.\r\n\r\nThe changes fully backward compatible and does not break any existing \"spark.executor.extraClassPath\" usages.\r\n\r\nThis patch was tested manually on our Hadoop cluster (4-nodes).\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/215246440","html_url":"https://github.com/apache/spark/pull/12678#issuecomment-215246440","issue_url":"https://api.github.com/repos/apache/spark/issues/12678","id":215246440,"user":{"login":"tgravescs","id":4563792,"avatar_url":"https://avatars.githubusercontent.com/u/4563792?v=3","gravatar_id":"","url":"https://api.github.com/users/tgravescs","html_url":"https://github.com/tgravescs","followers_url":"https://api.github.com/users/tgravescs/followers","following_url":"https://api.github.com/users/tgravescs/following{/other_user}","gists_url":"https://api.github.com/users/tgravescs/gists{/gist_id}","starred_url":"https://api.github.com/users/tgravescs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgravescs/subscriptions","organizations_url":"https://api.github.com/users/tgravescs/orgs","repos_url":"https://api.github.com/users/tgravescs/repos","events_url":"https://api.github.com/users/tgravescs/events{/privacy}","received_events_url":"https://api.github.com/users/tgravescs/received_events","type":"User","site_admin":false},"created_at":"2016-04-27T22:09:01Z","updated_at":"2016-04-27T22:09:01Z","body":"@vanzin \r\n\r\nIf --jars doesn't add it to the classpath then we broke it.\r\n\r\n --jars JARS                 Comma-separated list of local jars to include on the driver\r\n                              and executor classpaths.\r\n\r\nWhy is --files better?  Both go through distributed cache the same."}}