{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13712","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13712/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13712/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13712/events","html_url":"https://github.com/apache/spark/pull/13712","id":160722260,"number":13712,"title":"[SPARK-15990][YARN] Add rolling log aggregation support for Spark on yarn","user":{"login":"jerryshao","id":850797,"avatar_url":"https://avatars.githubusercontent.com/u/850797?v=3","gravatar_id":"","url":"https://api.github.com/users/jerryshao","html_url":"https://github.com/jerryshao","followers_url":"https://api.github.com/users/jerryshao/followers","following_url":"https://api.github.com/users/jerryshao/following{/other_user}","gists_url":"https://api.github.com/users/jerryshao/gists{/gist_id}","starred_url":"https://api.github.com/users/jerryshao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerryshao/subscriptions","organizations_url":"https://api.github.com/users/jerryshao/orgs","repos_url":"https://api.github.com/users/jerryshao/repos","events_url":"https://api.github.com/users/jerryshao/events{/privacy}","received_events_url":"https://api.github.com/users/jerryshao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-06-16T17:57:06Z","updated_at":"2016-06-23T14:46:03Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13712","html_url":"https://github.com/apache/spark/pull/13712","diff_url":"https://github.com/apache/spark/pull/13712.diff","patch_url":"https://github.com/apache/spark/pull/13712.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nYarn supports rolling log aggregation since 2.6, previously log will only be aggregated to HDFS after application is finished, it is quite painful for long running applications like Spark Streaming, thriftserver. Also out of disk problem will be occurred when log file is too large. So here propose to add support of rolling log aggregation for Spark on yarn.\r\n\r\nOne limitation for this is that log4j should be set to change to file appender, now in Spark itself uses console appender by default, in which file will not be created again once removed after aggregation. But I think lots of production users should have changed their log4j configuration instead of default on, so this is not a big problem.\r\n\r\n## How was this patch tested?\r\n\r\nManually verified with Hadoop 2.7.1.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/228073270","html_url":"https://github.com/apache/spark/pull/13712#issuecomment-228073270","issue_url":"https://api.github.com/repos/apache/spark/issues/13712","id":228073270,"user":{"login":"tgravescs","id":4563792,"avatar_url":"https://avatars.githubusercontent.com/u/4563792?v=3","gravatar_id":"","url":"https://api.github.com/users/tgravescs","html_url":"https://github.com/tgravescs","followers_url":"https://api.github.com/users/tgravescs/followers","following_url":"https://api.github.com/users/tgravescs/following{/other_user}","gists_url":"https://api.github.com/users/tgravescs/gists{/gist_id}","starred_url":"https://api.github.com/users/tgravescs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgravescs/subscriptions","organizations_url":"https://api.github.com/users/tgravescs/orgs","repos_url":"https://api.github.com/users/tgravescs/repos","events_url":"https://api.github.com/users/tgravescs/events{/privacy}","received_events_url":"https://api.github.com/users/tgravescs/received_events","type":"User","site_admin":false},"created_at":"2016-06-23T14:46:03Z","updated_at":"2016-06-23T14:46:03Z","body":"other then the minor doc changes it looks good. It might be nice to test compile against older version of hadoop just as test."}}
{"actor":{"display_login":"tgravescs"}}