<p>I'm trying to submit a Spark job on Kubernetes and write logs to S3. I'm using EKS and Spark client mode</p>

<p>I can write my Spark logs to a local directory, e.g., the below works:</p>

<pre><code>bin/spark-submit \
--master k8s://https://&lt;k8s-master&gt;:443 \
--deploy-mode client \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--conf spark.eventLog.enabled=true \
--conf spark.eventLog.dir=file:/tmp/spark_data \
--conf spark.kubernetes.container.image=gcav66/spark:spark-gus \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
file:///home/gus/spark-2.4.4-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.4.jar 
</code></pre>

<p>I'd like to store the logs in an S3 bucket. Based on the Spark docs (<a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#secret-management" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/running-on-kubernetes.html#secret-management</a>), I'm trying to pass my AWS Access and Secret keys via <code>spark.kubernetes.driver.secretKeyRef.[EnvName]</code></p>

<pre><code>bin/spark-submit \
--master k8s://https://&lt;k8s-master&gt;:443 \
--deploy-mode client \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=3 \
--conf spark.eventLog.enabled=true \
--conf spark.eventLog.dir=s3a://&lt;my-bucket&gt; \
--conf spark.history.fs.logDirectory=s3a://&lt;my-bucket&gt; \
--conf spark.kubernetes.container.image=gcav66/spark:spark-gus \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--spark.kubernetes.driver.secretKeyRef.AWS_ACCESS_KEY_ID=aws-secrets:key \
--spark.kubernetes.driver.secretKeyRef.AWS_SECRET_ACCESS_KEY=aws-secrets:secret \
--spark.kubernetes.executor.secretKeyRef.AWS_ACCESS_KEY_ID=aws-secrets:key \
--spark.kubernetes.executor.secretKeyRef.AWS_SECRET_ACCESS_KEY=aws-secrets:secret \
file:///home/gus/spark-2.4.4-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.4.jar 
</code></pre>

<p>The error I get is:
<code>Error: Unrecognized option: --spark.kubernetes.driver.secretKeyRef.AWS_ACCESS_KEY_ID</code></p>

<p>I have both an environment variable <code>AWS_ACCESS_KEY_ID</code> and I created a Kubernetes secret via <code>kubectl create secret generic aws-secrets --from-file=aws-access-key --from-file=aws-secret-key</code></p>

<pre><code>$ kubectl get secret
NAME                                                    TYPE                                  DATA   AGE
aws-secrets                                             Opaque                                2      8h
</code></pre>

<p>I don't think I have a typo - any ideas?</p>

grcavanaugh
<p>You need to pass the configs with <code>--conf</code>:</p>

<p>You do: <code>--spark.kubernetes.driver.secretKeyRef.AWS_ACCESS_KEY_ID=aws-secrets:key</code></p>

<p>You need: <code>--conf spark.kubernetes.driver.secretKeyRef.AWS_ACCESS_KEY_ID=aws-secrets:key</code></p>
