title: spark-submit in cluster deploy mode get application id to console
body: <p>I am stuck in one problem which I need to resolve quickly. I have gone through many posts and tutorial about spark cluster deploy mode, but I am clueless about the approach as I am stuck for some days.</p>
  
  <p>My use-case :- I have lots of spark jobs submitted using 'spark2-submit' command and I need to get the application id printed in the console once they are submitted. The spark jobs are submitted using cluster deploy mode. ( In normal client mode , its getting printed )</p>
  
  <p>Points I need to consider while creating solution :- I am not supposed to change code ( as it would take long time, cause there are many applications running  ), I can only provide log4j properties or some custom coding.</p>
  
  <p>My approach:-</p>
  
  <p>1) I have tried changing the log4j levels and various log4j parameters but the logging still goes to the centralized log directory.</p>
  
  <p>Part from my log4j.properties:-</p>
  
  <pre><code>log4j.logger.org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend=ALL,console
  log4j.appender.org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend.Target=System.out
  
  log4j.logger.org.apache.spark.deploy.SparkSubmit=ALL
  log4j.appender.org.apache.spark.deploy.SparkSubmit=console
  
  log4j.logger.org.apache.spark.deploy.SparkSubmit=TRACE,console
  log4j.additivity.org.apache.spark.deploy.SparkSubmit=false
  
  log4j.logger.org.apache.spark.deploy.yarn.Client=ALL
  log4j.appender.org.apache.spark.deploy.yarn.Client=console
  
  
  log4j.logger.org.apache.spark.SparkContext=WARN
  log4j.logger.org.apache.spark.scheduler.DAGScheduler=INFO,console
  
  log4j.logger.org.apache.hadoop.ipc.Client=ALL
  </code></pre>
  
  <p>2) I have also tried to add custom listener and I am able to get the spark application id after the applications finishes , but not to console.</p>
  
  <p>Code logic :-</p>
  
  <pre><code>public void onApplicationEnd(SparkListenerApplicationEnd arg0) 
      {
           for (Thread t : Thread.getAllStackTraces().keySet()) 
           {
              if (t.getName().equals("main"))
              {
                  System.out.println("The current state : "+t.getState());
  
                  Configuration config = new Configuration();
  
                  ApplicationId appId = ConverterUtils.toApplicationId(getjobUId);
  
                  // some logic to write to communicate with the main thread to print the app id to console.
              }
           }
      }
  </code></pre>
  
  <p>3) I have enabled the spark.eventLog to true and specified a directory in HDFS to write the event logs from spark-submit command . </p>
  
  <p>If anyone could help me in finding an approach to the solution, it would be really helpful. Or if I am doing something very wrong, any insights would help me.</p>
  
  <p>Thanks.</p>