{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11885","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11885/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11885/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11885/events","html_url":"https://github.com/apache/spark/pull/11885","id":142601510,"number":11885,"title":"[SPARK-14062][Yarn] Fix log4j and upload metrics.properties automatically with distributed cache","user":{"login":"jerryshao","id":850797,"avatar_url":"https://avatars.githubusercontent.com/u/850797?v=3","gravatar_id":"","url":"https://api.github.com/users/jerryshao","html_url":"https://github.com/jerryshao","followers_url":"https://api.github.com/users/jerryshao/followers","following_url":"https://api.github.com/users/jerryshao/following{/other_user}","gists_url":"https://api.github.com/users/jerryshao/gists{/gist_id}","starred_url":"https://api.github.com/users/jerryshao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerryshao/subscriptions","organizations_url":"https://api.github.com/users/jerryshao/orgs","repos_url":"https://api.github.com/users/jerryshao/repos","events_url":"https://api.github.com/users/jerryshao/events{/privacy}","received_events_url":"https://api.github.com/users/jerryshao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2016-03-22T10:07:46Z","updated_at":"2016-03-24T03:46:04Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11885","html_url":"https://github.com/apache/spark/pull/11885","diff_url":"https://github.com/apache/spark/pull/11885.diff","patch_url":"https://github.com/apache/spark/pull/11885.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n1. Currently log4j which uses distributed cache only adds to AM's classpath, not executor's, this is introduced in #9118, which breaks the original meaning of that PR, so here add log4j file to the classpath of both AM and executors.\r\n2. Automatically upload metrics.properties to distributed cache, so that it could be used by remote driver and executors implicitly.\r\n\r\n## How was this patch tested?\r\n\r\nUnit test and integration test is done.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/200644697","html_url":"https://github.com/apache/spark/pull/11885#issuecomment-200644697","issue_url":"https://api.github.com/repos/apache/spark/issues/11885","id":200644697,"user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars.githubusercontent.com/u/1694083?v=3","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"created_at":"2016-03-24T03:46:04Z","updated_at":"2016-03-24T03:46:04Z","body":"As I've said above, spark does *not* use the Hadoop configuration from the classpath in the executors. It uses the hadoop configuration broadcast from the driver.\r\n\r\nSo no matter what you add to the executor's classpath, it *will not* be used.\r\n\r\nAnd in any case, using the configuration present in the submitting node is more correct than using whatever configuration might or might not be available on the cluster nodes, which was the whole point of uploading the configuration archive to the AM in the first place."}}