title: YARN log aggregation on AWS EMR - UnsupportedFileSystemException
body: <p>I am struggling to enable YARN log aggregation for my Amazon EMR cluster.  I am following this documentation for the configuration: </p>
  
  <p><a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-debugging.html#emr-plan-debugging-logs-archive">http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-debugging.html#emr-plan-debugging-logs-archive</a></p>
  
  <p>Under the section titled: "To aggregate logs in Amazon S3 using the AWS CLI".</p>
  
  <p>I've verified that the hadoop-config bootstrap action puts the following in <code>yarn-site.xml</code></p>
  
  <pre><code>&lt;property&gt;&lt;name&gt;yarn.log-aggregation-enable&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt;
  &lt;property&gt;&lt;name&gt;yarn.log-aggregation.retain-seconds&lt;/name&gt;&lt;value&gt;-1&lt;/value&gt;&lt;/property&gt;
  &lt;property&gt;&lt;name&gt;yarn.log-aggregation.retain-check-interval-seconds&lt;/name&gt;&lt;value&gt;3000&lt;/value&gt;&lt;/property&gt;
  &lt;property&gt;&lt;name&gt;yarn.nodemanager.remote-app-log-dir&lt;/name&gt;&lt;value&gt;s3://mybucket/logs&lt;/value&gt;&lt;/property&gt;
  </code></pre>
  
  <p>I can run a sample job (<code>pi</code> from <code>hadoop-examples.jar</code>) and see that it completed successfully on the ResourceManager's GUI.</p>
  
  <p>It even creates a folder under <code>s3://mybucket/logs</code> named with the application id.  But the folder is empty, and if I run <code>yarn logs -applicationID &lt;applicationId&gt;</code>, I get a stacktrace:</p>
  
  <pre><code>14/10/20 23:02:15 INFO client.RMProxy: Connecting to ResourceManager at /10.XXX.XXX.XXX:9022
  Exception in thread "main" org.apache.hadoop.fs.UnsupportedFileSystemException: No AbstractFileSystem for scheme: s3
      at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:154)
      at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:242)
      at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:333)
      at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:330)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:330)
      at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:322)
      at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:85)
      at org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1388)
      at org.apache.hadoop.yarn.logaggregation.LogCLIHelpers.dumpAllContainersLogs(LogCLIHelpers.java:112)
      at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:137)
      at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:199) 
  </code></pre>
  
  <p>Which is doesn't make any sense to me; I can run <code>hdfs dfs -ls s3://mybucket/</code> and it lists the contents just fine.  The machines are getting credentials from AWS IAM Roles, I've tried adding fs.s3n.awsAccessKeyId and such to <code>core-site.xml</code> with no change in behavior.</p>
  
  <p>Any advice is much appreciated.</p>