<p>I am trying to run a sample spark job in kubernetes by following the steps mentioned here: <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/running-on-kubernetes.html</a>.</p>

<p>I am trying to send the spark driver and executor logs to Splunk. 
Does spark provide any configuration to do the same? 
How do I send the Splunk configurations like the HEC endpoint, port, token, etc in the spark-submit command?</p>

<p>I did try passing it as args to the the spark driver as </p>

<pre class="lang-sh prettyprint-override"><code>bin/spark-submit
  --deploy-mode cluster
  --class org.apache.spark.examples.JavaSparkPi
  --master k8s://http://127.0.0.1:8001
  --conf spark.executor.instances=2
  --conf spark.app.name=spark-pi
  --conf spark.kubernetes.container.image=gcr.io/spark-operator/spark:v2.4.4
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=&lt;account&gt;
  --conf spark.kubernetes.docker.image.pullPolicy=Always
  --conf spark.kubernetes.namespace=default
  local:///opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar
  --log-driver=splunk
  --log-opt splunk-url=&lt;url:port&gt;
  -—log-opt splunk-token=&lt;token&gt;
  --log-opt splunk-index=&lt;index&gt;
  --log-opt splunk-sourcetype=&lt;sourceType&gt;
  --log-opt splunk-format=json
</code></pre>

<p>but the logs were not forwarded to the desired index.</p>

<p>I am using spark version 2.4.4 to run spark-submit.</p>

<p>Thanks in advance for any inputs!!</p>

bornidiot
<p>Hi and welcome to the Stackoverflow.</p>

<p>I've searched the web for a while trying to find the similar to your question cases of <code>Spark + Splunk</code> usages. What I've managed to realize is that possibly you're mixing several things. Referring the Docker docs about <a href="https://docs.docker.com/config/containers/logging/splunk/" rel="nofollow noreferrer">Splunk logging driver</a> seems that you try to reproduce the same steps with `spark-submit. Unfortunately for you it doesn't work so.</p>

<p>Basically all the config options after <code>local:///opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar ...</code> in your script are the program arguments for the <code>org.apache.spark.examples.JavaSparkPi#main</code> method , which (unless you customize it) simply ignores them.</p>

<p>What you need to do is to connect your Kubrnetes cluster to the Splunk API. One of the ways of doing that is installing the <a href="https://github.com/splunk/splunk-connect-for-kubernetes" rel="nofollow noreferrer">Splunk Connector</a> to you Kubernetes cluster. Depending on your environment specifics there can be other ways of doing that, but reading the docs is a good place to start.</p>

<p>Hope it directs you to the right road.</p>
