<p>Im trying to run spark-submit to kubernetes cluster with spark 2.3 docker container image</p>

<p>The challenge im facing is application have a mainapplication.jar and other dependency files &amp; jars which are located in Remote location like AWS s3 ,but as per spark 2.3 documentation there is something called kubernetes init-container to download remote dependencies but in this case im not creating any Podspec to include init-containers in kubernetes, as per documentation Spark 2.3 spark/kubernetes internally creates Pods (driver,executor) So not sure how can i use init-container for spark-submit when there are remote dependencies.</p>

<p><a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-remote-dependencies" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-remote-dependencies</a></p>

<p>Please suggest</p>

<p>It works as it should with s3a:// urls. Unfortunatly getting s3a running on the stock spark-hadoop2.7.3 is problematic (authentication mainly), so I opted for building spark with Hadoop 2.9.1, since S3A has seen significant development there</p>

<p>I have created a <a href="https://gist.github.com/jalkjaer/4621d9034c5588880d854290c588004e" rel="nofollow noreferrer">gist</a> with the steps needed to </p>

<ul>
<li>build spark with new hadoop dependencies </li>
<li>build the docker image for k8s</li>
<li>push image to ECR</li>
</ul>

<p>The script also creates a second docker image with the S3A dependencies added and base conf settings for enabling S3A using IAM credentials so running in AWS doesn't require putting access/secretkey in conf files/args</p>

<p>I havn't run any production spark jobs yet using the image, but have tested that basic saving and loading to s3a urls does work. </p>

<p>I have yet to experiment with <a href="https://hadoop.apache.org/docs/r2.9.1/hadoop-aws/tools/hadoop-aws/s3guard.html" rel="nofollow noreferrer">S3Guard</a> which uses DynamoDB to ensure that S3 writes/reads are consistent - similarly to EMRFS</p>
