<p>I am trying to use kuberenets as cluster manger for <code>spark</code>. I also want to ship the container logs to <code>splunk</code>. Now I do have <strong>monitoring stack</strong> deployed (fluent-bit, prometheus etc)in the same namespace and the way it works is if your <code>pod</code> has a certain <code>environment_variable</code> it will start reading the logs and push it to <strong>splunk</strong>.
The thing I am not able to find is how do I set a environment variable and populate it</p>

<pre><code>bin/spark-submit \
   --deploy-mode cluster \
   --class org.apache.spark.examples.SparkPi \
   --master k8s://https://my-kube-cluster.com \
   --conf spark.executor.instances=2 \
   --conf spark.app.name=spark-pi \
   ....
   ....
   ....
   --conf spark.kubernetes.driverEnv.UID="set it to spark driver pod id" \
   local:///opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar
</code></pre>

devnull
<p>To configure additional Spark Driver Pod environment variables you can pass additional <code>--conf spark.kubernetes.driverEnv.EnvironmentVariableName=EnvironmentVariableValue</code> (please refer <a href="http://spark.apache.org/docs/2.4.5/running-on-kubernetes.html#spark-properties" rel="nofollow noreferrer">docs</a> for more details).</p>

<p>To configure additional Spark Executor Pods environment variables you can pass additional <code>--conf spark.executorEnv.EnvironmentVariableName=EnvironmentVariableValue</code> (please refer <a href="https://spark.apache.org/docs/latest/configuration.html#runtime-environment" rel="nofollow noreferrer">docs</a> for more details).</p>

<p>Hope it helps.</p>
