{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12203","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12203/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12203/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12203/events","html_url":"https://github.com/apache/spark/pull/12203","id":146259125,"number":12203,"title":"[SPARK-14423][YARN] Avoid same name files added to distributed cache again","user":{"login":"jerryshao","id":850797,"avatar_url":"https://avatars.githubusercontent.com/u/850797?v=3","gravatar_id":"","url":"https://api.github.com/users/jerryshao","html_url":"https://github.com/jerryshao","followers_url":"https://api.github.com/users/jerryshao/followers","following_url":"https://api.github.com/users/jerryshao/following{/other_user}","gists_url":"https://api.github.com/users/jerryshao/gists{/gist_id}","starred_url":"https://api.github.com/users/jerryshao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerryshao/subscriptions","organizations_url":"https://api.github.com/users/jerryshao/orgs","repos_url":"https://api.github.com/users/jerryshao/repos","events_url":"https://api.github.com/users/jerryshao/events{/privacy}","received_events_url":"https://api.github.com/users/jerryshao/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-04-06T10:00:13Z","updated_at":"2016-07-03T04:21:41Z","closed_at":"2016-04-18T17:15:41Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12203","html_url":"https://github.com/apache/spark/pull/12203","diff_url":"https://github.com/apache/spark/pull/12203.diff","patch_url":"https://github.com/apache/spark/pull/12203.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIn the current implementation of assembly-free spark deployment, jars under `assembly/target/scala-xxx/jars` will be uploaded to distributed cache by default, there's a chance these jars' name will be conflicted with name of jars specified in `--jars`, this will introduce exception when starting application:\r\n\r\n```\r\nclient token: N/A\r\n\t diagnostics: Application application_1459907402325_0004 failed 2 times due to AM Container for appattempt_1459907402325_0004_000002 exited with  exitCode: -1000\r\nFor more detailed output, check application tracking page:http://hw12100.local:8088/proxy/application_1459907402325_0004/Then, click on links to logs of each attempt.\r\nDiagnostics: Resource hdfs://localhost:8020/user/sshao/.sparkStaging/application_1459907402325_0004/avro-mapred-1.7.7-hadoop2.jar changed on src filesystem (expected 1459909780508, was 1459909782590\r\njava.io.IOException: Resource hdfs://localhost:8020/user/sshao/.sparkStaging/application_1459907402325_0004/avro-mapred-1.7.7-hadoop2.jar changed on src filesystem (expected 1459909780508, was 1459909782590\r\n\tat org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)\r\n\tat org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)\r\n\tat org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)\r\n\tat org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)\r\n\tat java.security.AccessController.doPrivileged(Native Method)\r\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\r\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\r\n\tat org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)\r\n\tat org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\r\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\nSo here by checking the name of file to avoid same name files uploaded again.\r\n\r\n## How was this patch tested?\r\n\r\nUnit test and manual integrated test is done locally.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/230134636","html_url":"https://github.com/apache/spark/pull/12203#issuecomment-230134636","issue_url":"https://api.github.com/repos/apache/spark/issues/12203","id":230134636,"user":{"login":"RicoGit","id":6422229,"avatar_url":"https://avatars.githubusercontent.com/u/6422229?v=3","gravatar_id":"","url":"https://api.github.com/users/RicoGit","html_url":"https://github.com/RicoGit","followers_url":"https://api.github.com/users/RicoGit/followers","following_url":"https://api.github.com/users/RicoGit/following{/other_user}","gists_url":"https://api.github.com/users/RicoGit/gists{/gist_id}","starred_url":"https://api.github.com/users/RicoGit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RicoGit/subscriptions","organizations_url":"https://api.github.com/users/RicoGit/orgs","repos_url":"https://api.github.com/users/RicoGit/repos","events_url":"https://api.github.com/users/RicoGit/events{/privacy}","received_events_url":"https://api.github.com/users/RicoGit/received_events","type":"User","site_admin":false},"created_at":"2016-07-03T04:21:41Z","updated_at":"2016-07-03T04:21:41Z","body":"Hi guys, it is possible to apply this patch to version 1.6? What can I do for this?"}}
{"actor":{"display_login":"RicoGit"}}