<p>We have deployed Apache Spark on Azure Kubernetes Services (AKS).</p>
<p>Able to submit spark application via CLI <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#cluster-mode" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/running-on-kubernetes.html#cluster-mode</a></p>
<p><strong>Question</strong>: Is it possible to submit a spark job/run a spark application from Azure Data factory version 2? That way we can orchestrate spark application from data factory.</p>

<h5>High-Level Architecture</h5>
<p><a href="https://i.stack.imgur.com/4QFj3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4QFj3.png" alt="enter image description here" /></a></p>
<p>Quick explanation of the architecture flow:</p>
<ul>
<li><p>In order to connect any on-premise data sources to Azure, you can install an integration runtime (executable installer from Data Factory) on a dedicated VM. This allows Data Factory to create connections to these on-premise servers.</p>
</li>
<li><p>A Data Factory pipeline will load raw data into Data Lake Storage Gen 2. ADLS2 applies hierarchical namespace to blob storage (think folders). A downstream task in the pipeline will trigger a custom Azure Function.</p>
</li>
<li><p>Custom python Azure Function will load a config yaml from ADLS2, and submit a spark application to k8s service via k8s python client.
The container registry is essentially a docker hub in Azure. Docker images are deployed to the registry, and k8s will pull as required.</p>
</li>
<li><p>Upon submission by the Azure Function, k8s will pull the spark image from container registry and execute the spark application. Spark leverages the k8s scheduler to automatically spin up driver and executor pods. Once the application is complete, the executor pods self-terminate while the driver pod persists logs and remains in &quot;completed&quot; state (which uses no resources).</p>
</li>
</ul>
<p>ADF Pipeline Example:</p>
<p><a href="https://i.stack.imgur.com/AgiAN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AgiAN.png" alt="enter image description here" /></a></p>
<p>few challenges with this setup:</p>
<ol>
<li>No programmatic way of submitting a spark application to k8s (running command line &quot;kubectl&quot; or &quot;spark-submit&quot; isn't going to cut it in production)</li>
<li>No OOTB method to orchestrate a spark application submission to k8s using Data Factory</li>
<li>Spark 2.4.5 with Hadoop 2.7 doesn't support read/writes to ADLS2, and cannot be built with Hadoop 3.2.1 (unless you have extensive dev knowledge of spark source code)</li>
</ol>
<p>Let's walk through the top secret tools used to make the magic happen. Deployed a custom <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator" rel="nofollow noreferrer">spark-on-k8s-operator</a> resource to the kubernetes cluster which allows submitting spark applications with a <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-py-pi.yaml" rel="nofollow noreferrer">yaml</a> file. However, the documentation only shows how to submit a spark app using cmd line <code>kubectl apply -f yaml</code>. To submit the spark application programmatically (ie - via REST API), leveraged the <a href="https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/CustomObjectsApi.md" rel="nofollow noreferrer">CustomObjectsApi</a> from k8s python client SDK inside a python Azure Function. Why? Because ADF has an OOTB task to trigger Azure Functions. 🎉</p>
<p>Spark 3.0.0-preview2 already has built-in integration with Hadoop 3.2+ so that's not so top secret, but there are a couple of things to look out for when you build the docker image. The bin/docker-image-tool.sh needs extra quotes on line 153 (I think this is just a problem with windows filesystem). To support read/write to ADLS2, you need to download the <a href="https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure" rel="nofollow noreferrer">hadoop-azure jar</a> &amp; <a href="https://mvnrepository.com/artifact/org.wildfly.openssl/wildfly-openssl" rel="nofollow noreferrer">wildfly.openssl</a> jar (place them in spark_home/jars). Finally, replace the kubernetes/dockerfiles/spark/entrypoint.sh with the one from Spark 2.4.5 pre-built for Hadoop 2.7+ (missing logic to support python driver).</p>
<p>Quick tips: package any custom jars into spark_home/jars before building your docker image &amp; reference them as dependencies via &quot;local:///opt/spark/jars&quot;, upload extra python libs to ADLS2 and use <code>sc.addPyFile(public_url_of_python_lib)</code> in your main application before importing.</p>
<p>Reference: <a href="https://www.linkedin.com/pulse/ultimate-evolution-spark-data-pipelines-azure-kubernetes-kenny-bui/" rel="nofollow noreferrer">https://www.linkedin.com/pulse/ultimate-evolution-spark-data-pipelines-azure-kubernetes-kenny-bui/</a></p>
