title: Spark - How to colourise terminal output from spark-submit
body: <p>When I run <a href="https://spark.apache.org/docs/latest/submitting-applications.html#launching-applications-with-spark-submit" rel="nofollow noreferrer">spark-submit</a> it works successfully, but the output is not colourised.</p>
  <pre><code>(/Users/me/bai/conda-envs/spark-mllib-kmeans) me@my-mbp spark-mllib-kmeans % spark-submit spark-helloWorld.py
  WARNING: An illegal reflective access operation has occurred
  WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.0.1/libexec/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
  WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
  WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
  WARNING: All illegal access operations will be denied in a future release
  20/12/22 12:18:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  20/12/22 12:18:34 INFO SparkContext: Running Spark version 3.0.1
  20/12/22 12:18:34 INFO ResourceUtils: ==============================================================
  20/12/22 12:18:34 INFO ResourceUtils: Resources for spark.driver:
  
  20/12/22 12:18:34 INFO ResourceUtils: ==============================================================
  20/12/22 12:18:34 INFO SparkContext: Submitted application: Simple App
  ...
  
  </code></pre>
  <p>I am using Spark version 3.0.1:</p>
  <pre><code>(base) me@my-mbp spark-mllib-kmeans % spark-shell --version
  WARNING: An illegal reflective access operation has occurred
  WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.0.1/libexec/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
  WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
  WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
  WARNING: All illegal access operations will be denied in a future release
  Welcome to
        ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /___/ .__/\_,_/_/ /_/\_\   version 3.0.1
        /_/
                          
  Using Scala version 2.12.10, OpenJDK 64-Bit Server VM, 14.0.1
  Branch HEAD
  Compiled by user ubuntu on 2020-08-28T08:58:35Z
  Revision 2b147c4cd50da32fe2b4167f97c8142102a0510d
  Url https://gitbox.apache.org/repos/asf/spark.git
  Type --help for more information.
  
  </code></pre>
  <p>I am using the default Mac terminal program on latest Mac OS:</p>
  <pre><code>% uname -a
  Darwin my-mbp.lan 20.2.0 Darwin Kernel Version 20.2.0: Wed Dec  2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64 x86_64
  </code></pre>
  <p>I would like to see the different log statement levels (WARN/INFO/ERROR) in different colours. Perhaps other use of colours to differentiate the output from spark framework and the output from my application.</p>
  <p>Given there is so much framework level output and there is noise of WARNINGS due to framework issues, I was hoping better use of colour could help me to scan my output quicker.</p>
  <p>Is there a simple solution for this?</p>
  <p>I see this behaviour in both native Mac Terminal and MS VSC integrated terminal.</p>
  <p>I saw the output line:</p>
  <pre><code>Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  </code></pre>
  <p>I see I can create then edit the log4j config file here, so perhaps I just need the config for the log4j config file to colourise all output.</p>
  <pre><code>% cd $SPARK_HOME/conf                          
  % cp log4j.properties.template log4j.properties
  </code></pre>