title: Spark Streaming & centralized logging: add executorId to logs
body: <p>I am trying to set up centralized logging for a Spark streaming application. My specific use case is to export driver &amp; executors logs to AWS CloudWatch logs but I think part of my question is agnostic to CloudWatch logs and EMR.</p>
  
  <p>Thanks to <a href="https://spark.apache.org/docs/latest/running-on-yarn.html" rel="nofollow noreferrer">Running Spark on Yarn</a> / Debugging your Application, I know how to pass in my own log4j configuration file, and even that <code>spark.yarn.app.container.log.dir</code> is available. I am already able to make each executor logs into a local file and to run CloudWatch logs agent on each instance to publish the local file into CloudWatch logs.</p>
  
  <p>The issue with this setup is that logs from all executors are mixed. It is not possible to know which executor produced which event. I had like to add something like the executor id to CloudWatch stream name or directly into to logs. However I don't see how to inject that information from Spark to log4j. </p>
  
  <p>I gave a quick look to <code>ExecutorRunnable.scala</code> but failed to spot a suitable environment variable or system property. Are you aware of any property that could fit my need? How do people manage this issue? (even on an ELK stack or whatever)</p>