title: When use oozie spark action, spark driver spawned on one Node can find custom log4j config file, but on other Nodes cannot find. why?
body: <p>Use oozie to run spark action workflow has problem.
  If driver spawned on node (172.12.0.27), log config is <strong>always right</strong>.
  If driver spawned on other nodes(172.12.0.18,172.12.0.20), log config is <strong>always wrong</strong>. </p>
  
  <p>Use spark-submit to run job without above problem, driver spawned on every node works right. </p>
  
  <p>How can I track problem? </p>
  
  <p>There are three nodes(172.12.0.27,172.12.0.18,172.12.0.20). If use spark-submit with custom log4j config file, no problem on any of the node .</p>
  
  <p><strong>spark-submit works right.</strong></p>
  
  <pre class="lang-sh prettyprint-override"><code>spark-submit --master yarn --deploy-mode cluster --driver-memory 1g --num-executors 4 --executor-memory 1g --files "/root/alenym/log4j.properties" --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties -Dtb.spark.prod.env=true" --class com.stc.data.thingsboard.jobs.example.TestLogJob /root/alenym/dp_advance_analysis/bigdata/tb-sql-analysis/target/tb-sql-analysis-1.0.0-SNAPSHOT.jar 
  </code></pre>
  
  <p><strong>oozie workflow has wrong.</strong></p>
  
  <ol>
  <li>workflow.xml like this below.</li>
  </ol>
  
  <pre class="lang-xml prettyprint-override"><code>&lt;workflow-app xmlns='uri:oozie:workflow:0.5' name='spark-test'&gt;
      &lt;start to='spark-node' /&gt;
  
      &lt;action name='spark-node'&gt;
          &lt;spark xmlns="uri:oozie:spark-action:0.1"&gt;
              &lt;job-tracker&gt;${jobTracker}&lt;/job-tracker&gt;
              &lt;name-node&gt;${nameNode}&lt;/name-node&gt;
          &lt;job-xml&gt;${nameNode}/user/${wf:user()}/${examplesRoot}/spark/hive-site.xml&lt;/job-xml&gt;
              &lt;master&gt;${master}&lt;/master&gt;
          &lt;mode&gt;cluster&lt;/mode&gt;
              &lt;name&gt;spark-test-oozie&lt;/name&gt;
              &lt;class&gt;com.stc.data.thingsboard.jobs.example.TestLogJob&lt;/class&gt;
              &lt;jar&gt;${nameNode}/user/${wf:user()}/${examplesRoot}/spark/lib/tb-sql-analysis-1.0.0-SNAPSHOT.jar&lt;/jar&gt;
          &lt;spark-opts&gt;--driver-memory 1g --num-executors 10 --executor-memory 1g --conf spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j-ym.properties -Dtb.spark.prod.env=true &lt;/spark-opts&gt;
          &lt;/spark&gt;
          &lt;ok to="end" /&gt;
          &lt;error to="fail" /&gt;
      &lt;/action&gt;
  
      &lt;kill name="fail"&gt;
          &lt;message&gt;Workflow failed, error
              message[${wf:errorMessage(wf:lastErrorNode())}]
          &lt;/message&gt;
      &lt;/kill&gt;
      &lt;end name='end' /&gt;
  &lt;/workflow-app&gt;
  
  </code></pre>
  
  <ol start="2">
  <li>job.properties</li>
  </ol>
  
  <pre><code>nameNode=hdfs://HDFS80599
  jobTracker=rm1
  master=yarn
  queueName=default
  examplesRoot=batchtest
  oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/spark
  </code></pre>
  
  <ol start="3">
  <li>custom "log4j-ym.properties" file is in "./lib" directory, so that --files has include</li>
  </ol>
  
  <p><strong>hdfs://HDFS80599/user/root/batchtest/spark/lib/log4j-ym.properties#log4j-ym.properties,</strong></p>
  
  <pre><code>
  Log Type: stdout
  
  Log Upload Time: Mon Aug 26 10:05:44 +0800 2019
  
  Log Length: 261750
  
  Oozie Launcher starts
  
  
  
  Oozie Java/Map-Reduce/Pig action launcher-job configuration
  =================================================================
  Workflow job id   : 0000993-190728182827383-oozie-hado-W
  Workflow action id: 0000993-190728182827383-oozie-hado-W@spark-node
  
  Classpath         :
  ------------------------
  ...
  ...
  
  ------------------------
  
  Main class        : org.apache.oozie.action.hadoop.SparkMain
  
  Maximum output    : 2048
  
  Arguments         :
  
  Java System Properties:
  ------------------------
  #
  #Mon Aug 26 10:04:54 CST 2019
  java.runtime.name=Java(TM) SE Runtime Environment
  sun.boot.library.path=/usr/local/jdk/jre/lib/amd64
  java.vm.version=25.191-b12
  oozie.action.externalChildIDs=/data/emr/yarn/local/usercache/root/appcache/application_1565412953433_3795/container_e22_1565412953433_3795_01_000002/externalChildIDs
  hadoop.root.logger=INFO,CLA
  java.vm.vendor=Oracle Corporation
  java.vendor. url=/emr-yarn-jobhistory/http://172.21.0.48:5024/http\://java.oracle.com/
  path.separator=\:
  java.vm.name=Java HotSpot(TM) 64-Bit Server VM
  file.encoding.pkg=sun.io
  oozie.job.launch.time=1566785085000
  user.country=US
  sun.java.launcher=SUN_STANDARD
  sun.os.patch.level=unknown
  java.vm.specification.name=Java Virtual Machine Specification
  user.dir=/data/emr/yarn/local/usercache/root/appcache/application_1565412953433_3795/container_e22_1565412953433_3795_01_000002
  oozie.action.newId=/data/emr/yarn/local/usercache/root/appcache/application_1565412953433_3795/container_e22_1565412953433_3795_01_000002/newId
  java.runtime.version=1.8.0_191-b12
  java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment
  java.endorsed.dirs=/usr/local/jdk/jre/lib/endorsed
  os.arch=amd64
  oozie.job.id=0000993-190728182827383-oozie-hado-W
  oozie.action.id=0000993-190728182827383-oozie-hado-W@spark-node
  yarn.app.container.log.dir=/data/emr/yarn/logs/application_1565412953433_3795/container_e22_1565412953433_3795_01_000002
  java.io.tmpdir=./tmp
  ...
  ...
  &gt;&gt;&gt; Invoking Main class now &gt;&gt;&gt;
  
  Fetching child yarn jobs
  tag id : oozie-e1db067250aafedb4df7ee644cd82ab4
  Child yarn jobs are found - 
  Warning: Spark Log4J settings are overwritten. Child job IDs may not be available
  Spark Version 2.3
  Spark Action Main class        : org.apache.spark.deploy.SparkSubmit
  
  Oozie Spark action configuration
  =================================================================
  
                      --master
                      yarn
                      --deploy-mode
                      cluster
                      --name
                      spark-test-oozie
                      --class
                      com.stc.data.thingsboard.jobs.example.TestLogJob
                      --conf
                      spark.oozie.action.id=0000993-190728182827383-oozie-hado-W@spark-node
                      --conf
                      spark.oozie.child.mapreduce.job.tags=oozie-e1db067250aafedb4df7ee644cd82ab4
                      --conf
                      spark.oozie.action.rootlogger.log.level=INFO
                      --conf
                      spark.oozie.job.id=0000993-190728182827383-oozie-hado-W
                      --conf
                      spark.oozie.action.spark.setup.hadoop.conf.dir=false
                      --conf
                      spark.oozie.HadoopAccessorService.created=true
                      --driver-memory
                      1g
                      --num-executors
                      10
                      --executor-memory
                      1g
                      --conf
                      spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j-ym.properties -Dtb.spark.prod.env=true
                      --conf
                      spark.executor.extraClassPath=$PWD/*
                      --conf
                      spark.driver.extraClassPath=$PWD/*
                      --conf
                      spark.yarn.tags=oozie-e1db067250aafedb4df7ee644cd82ab4
                      --conf
                      spark.yarn.security.tokens.hadoopfs.enabled=false
                      --conf
                      spark.yarn.security.tokens.hive.enabled=false
                      --conf
                      spark.yarn.security.tokens.hbase.enabled=false
                      --conf
                      spark.yarn.security.credentials.hadoopfs.enabled=false
                      --conf
                      spark.yarn.security.credentials.hive.enabled=false
                      --conf
                      spark.yarn.security.credentials.hbase.enabled=false
                      --conf
                      spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
                      --files
                      hdfs://HDFS80599/user/root/batchtest/spark/lib/breeze_2.11-0.13.2.jar,...,hdfs://HDFS80599/user/root/batchtest/spark/lib/calcite-core-1.2.0-incubating.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/calcite-linq4j-1.2.0-incubating.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/chill-java-0.8.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/chill_2.11-0.8.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/commons-beanutils-1.7.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/commons-beanutils-core-1.8.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/commons-cli-1.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/commons-codec-1.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/commons-collections-3.2.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/commons-compiler-3.0.8.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/commons-compress-1.4.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/commons-compress-1.9.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/commons-configuration-1.6.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/commons-crypto-1.0.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/datanucleus-core-4.1.17.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/datanucleus-rdbms-4.1.19.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/derby-10.10.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/disruptor-3.3.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/eigenbase-properties-1.1.5.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/fastutil-6.5.6.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/findbugs-annotations-1.3.9-1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/flume-ng-configuration-1.6.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/flume-ng-core-1.6.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/flume-ng-sdk-1.6.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/geronimo-annotation_1.0_spec-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/geronimo-jaspic_1.0_spec-1.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/geronimo-jta_1.1_spec-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/gson-2.7.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/guava-11.0.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/guava-14.0.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/guice-3.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/guice-assistedinject-3.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/guice-servlet-3.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-common-2.8.4-tests.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-common-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-hdfs-2.8.4-tests.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-hdfs-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-hdfs-client-2.8.4-tests.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-hdfs-client-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-hdfs-native-client-2.8.4-tests.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-hdfs-native-client-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-hdfs-nfs-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-mapreduce-client-app-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-mapreduce-client-common-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-mapreduce-client-core-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-mapreduce-client-hs-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-mapreduce-client-hs-plugins-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-mapreduce-client-jobclient-2.8.4-tests.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-mapreduce-client-jobclient-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-mapreduce-client-shuffle-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-mapreduce-examples-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-nfs-2.8.4.jar,hdfs://HDFS80599/user/hadoop/share/lib/lib_20190728182750/oozie/hadoop-temrfs-1.0.6.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-api-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-applications-distributedshell-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-applications-unmanaged-am-launcher-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-client-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-common-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-registry-2.7.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-registry-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-server-applicationhistoryservice-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-server-common-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-server-nodemanager-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-server-resourcemanager-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-server-sharedcachemanager-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-server-tests-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-server-timeline-pluginstorage-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hadoop-yarn-server-web-proxy-2.8.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hbase-annotations-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hbase-client-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hbase-common-1.1.1-tests.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hbase-common-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hbase-hadoop-compat-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hbase-hadoop2-compat-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hbase-prefix-tree-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hbase-procedure-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hbase-protocol-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hbase-server-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hive-beeline-1.2.1.spark2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hive-cli-1.2.1.spark2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hive-exec-1.2.1.spark2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hive-hbase-handler-2.3.3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hive-hcatalog-core-2.3.3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hive-jdbc-1.2.1.spark2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hive-metastore-1.2.1.spark2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hk2-api-2.4.0-b34.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hk2-locator-2.4.0-b34.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/hk2-utils-2.4.0-b34.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/htrace-core-3.0.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/htrace-core-3.1.0-incubating.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/httpclient-4.3.6.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/httpcore-4.3.3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/ivy-2.4.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jackson-annotations-2.6.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jackson-annotations-2.6.5.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jackson-core-2.6.5.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jackson-core-asl-1.9.13.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jackson-databind-2.6.5.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jackson-jaxrs-1.9.13.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jackson-mapper-asl-1.9.13.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jackson-module-paranamer-2.6.5.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jackson-module-scala_2.11-2.6.5.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jackson-xc-1.9.13.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jamon-runtime-2.3.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/janino-3.0.8.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jasper-compiler-5.5.23.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jasper-runtime-5.5.23.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/java-xmlbuilder-0.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/java-xmlbuilder-1.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/javassist-3.18.1-GA.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/javax.annotation-api-1.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/javax.inject-1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/javax.inject-2.4.0-b34.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/javax.jdo-3.2.0-m3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/javax.servlet-api-3.1.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/javax.ws.rs-api-2.0.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/javolution-5.5.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jaxb-api-2.2.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jaxb-impl-2.2.3-1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jcl-over-slf4j-1.7.16.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jcodings-1.0.8.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jcommander-1.30.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jdo-api-3.0.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jersey-client-1.9.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jersey-client-2.22.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jersey-common-2.22.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jersey-container-servlet-2.22.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jersey-container-servlet-core-2.22.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jersey-core-1.9.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jersey-guava-2.22.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jersey-json-1.9.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jersey-media-jaxb-2.22.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jersey-server-1.9.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jersey-server-2.22.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jets3t-0.9.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jets3t-0.9.3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jettison-1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jetty-6.1.14.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jetty-all-7.6.0.v20120127.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jetty-sslengine-6.1.26.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jetty-util-6.1.26.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jline-0.9.94.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jline-2.12.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/joda-time-2.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jodd-core-3.5.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/joni-2.1.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jpam-1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jsch-0.1.42.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/json-1.8.jar,hdfs://HDFS80599/user/hadoop/share/lib/lib_20190728182750/oozie/json-simple-1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/json4s-ast_2.11-3.2.11.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/json4s-core_2.11-3.2.11.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/json4s-jackson_2.11-3.2.11.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jsp-2.1-6.1.14.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jsp-api-2.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jsp-api-2.1-6.1.14.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jsr305-1.3.9.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jsr305-3.0.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jta-1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jtransforms-2.4.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/jul-to-slf4j-1.7.16.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/kafka-clients-0.8.2.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/kafka_2.11-0.8.2.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/kryo-shaded-3.0.3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/leveldbjni-all-1.8.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/libfb303-0.9.3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/libthrift-0.9.3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/log4j-1.2-api-2.6.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/log4j-1.2.17.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/log4j-api-2.6.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/log4j-core-2.6.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/log4j-web-2.6.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/log4j-ym.properties#log4j-ym.properties,hdfs://HDFS80599/user/root/batchtest/spark/lib/log4j2.xml#log4j2.xml,hdfs://HDFS80599/user/root/batchtest/spark/lib/lz4-java-1.4.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/machinist_2.11-0.6.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/macro-compat_2.11-1.1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/mail-1.4.7.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/metrics-core-2.2.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/metrics-core-3.1.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/metrics-graphite-3.1.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/metrics-json-3.1.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/metrics-json-3.1.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/metrics-jvm-3.1.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/metrics-jvm-3.1.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/mina-core-2.0.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/minlog-1.3.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/mx4j-3.0.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/objenesis-2.1.jar,hdfs://HDFS80599/user/hadoop/share/lib/lib_20190728182750/oozie/oozie-hadoop-utils-hadoop-2-4.3.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/oozie-sharelib-hive2-4.3.1.jar,hdfs://HDFS80599/user/hadoop/share/lib/lib_20190728182750/oozie/oozie-sharelib-oozie-4.3.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/oozie-sharelib-spark-4.3.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/opencsv-2.3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/orc-core-1.3.3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/oro-2.0.8.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/osgi-resource-locator-1.0.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/paranamer-2.3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/parquet-column-1.8.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/parquet-common-1.8.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/parquet-encoding-1.8.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/parquet-format-2.3.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/parquet-hadoop-1.8.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/parquet-hadoop-bundle-1.6.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/parquet-hadoop-bundle-1.8.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/parquet-jackson-1.8.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/pmml-model-1.2.15.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/pmml-schema-1.2.15.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/protobuf-java-2.5.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/py4j-0.10.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/pyrolite-4.13.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/scala-compiler-2.11.8.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/scala-library-2.11.8.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/scala-parser-combinators_2.11-1.0.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/scala-reflect-2.11.8.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/scala-xml_2.11-1.0.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/scalap-2.11.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/servlet-api-2.5-6.1.14.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/shapeless_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/slf4j-api-1.7.16.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/slf4j-log4j12-1.7.16.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/slider-core-0.90.2-incubating.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/snappy-java-1.0.5.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/snappy-java-1.1.2.6.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-catalyst_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-core_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-graphx_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-hive-thriftserver_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-hive_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-kvstore_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-launcher_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-mllib-local_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-mllib_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-network-common_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-network-shuffle_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-repl_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-sketch_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-sql_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-streaming_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-tags_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-unsafe_2.11-2.3.2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spire-macros_2.11-0.13.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/spire_2.11-0.13.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/stax-api-1.0-2.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/stream-2.7.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/stringtemplate-3.2.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/super-csv-2.2.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/tephra-api-0.6.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/tephra-core-0.6.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/tephra-hbase-compat-1.0-0.6.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/transaction-api-1.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/twill-api-0.6.0-incubating.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/twill-common-0.6.0-incubating.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/twill-core-0.6.0-incubating.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/twill-discovery-api-0.6.0-incubating.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/twill-discovery-core-0.6.0-incubating.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/twill-zookeeper-0.6.0-incubating.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/univocity-parsers-2.2.1.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/unused-1.0.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/validation-api-1.1.0.Final.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/xbean-asm5-shaded-4.4.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/xercesImpl-2.11.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/xml-apis-1.4.01.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/xmlenc-0.52.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/xz-1.0.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/zkclient-0.3.jar,hdfs://HDFS80599/user/root/batchtest/spark/lib/zookeeper-3.4.6.jar,spark-log4j.properties,hive-site.xml
                      --conf
                      spark.yarn.jars=hdfs://HDFS80599/user/root/batchtest/spark/lib/spark-yarn_2.11-2.3.2.jar
                      --verbose
                      hdfs://HDFS80599/user/root/batchtest/spark/lib/tb-sql-analysis-1.0.0-SNAPSHOT.jar
  </code></pre>
  
  <p>The <strong>expected</strong> result is below. can output utf-8 log:</p>
  
  <p><strong>TestLogJob:41 - This is from log.info|zhong wen (中文)</strong></p>
  
  <pre><code>Log Type: stderr
  
  Log Upload Time: Mon Aug 26 10:05:38 +0800 2019
  
  Log Length: 493
  
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/data/emr/yarn/local/filecache/0/34352/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/local/service/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  
  Log Type: stdout
  
  Log Upload Time: Mon Aug 26 10:05:38 +0800 2019
  
  Log Length: 286
  
  2019-08-26 10:05:05 INFO  AbstractTbSparkSqlJob:130 - tb.spark.prod.env=true
  2019-08-26 10:05:05 INFO  AbstractTbSparkSqlJob:131 - tb.spark.test.env=false
  2019-08-26 10:05:36 INFO  TestLogJob:41 - This is from log.info|zhong wen (中文)
  </code></pre>
  
  <p>The <strong>wrong</strong> log example like this below. <strong>Driver use Spark's default log4j profile: org/apache/spark/log4j-defaults.properties</strong></p>
  
  <p>and so that log content is not utf-8 :</p>
  
  <p><strong>TestLogJob:41 - This is from log.info|zhong wen (??)</strong></p>
  
  <pre><code>Log Type: stderr
  
  Log Upload Time: Mon Aug 26 10:05:37 +0800 2019
  
  Log Length: 571
  
  Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/data/emr/yarn/local/filecache/41582/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/local/service/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  
  Log Type: stdout
  
  Log Upload Time: Mon Aug 26 10:05:37 +0800 2019
  
  Log Length: 398
  
  2019-08-26 10:05:05.284 [Driver] INFO  com.stc.data.thingsboard.AbstractTbSparkSqlJob - tb.spark.prod.env=true
  2019-08-26 10:05:05.288 [Driver] INFO  com.stc.data.thingsboard.AbstractTbSparkSqlJob - tb.spark.test.env=false
  2019-08-26 10:05:36.599 [Driver] INFO  com.stc.data.thingsboard.jobs.example.TestLogJob - This is from log.info|zhong wen (??)
  </code></pre>