<p>I am deploying pyspark in my aks Kubernetes cluster using this guides:</p>
<ul>
<li><a href="https://towardsdatascience.com/ignite-the-spark-68f3f988f642" rel="nofollow noreferrer">https://towardsdatascience.com/ignite-the-spark-68f3f988f642</a></li>
<li><a href="http://blog.brainlounge.de/memoryleaks/getting-started-with-spark-on-kubernetes/" rel="nofollow noreferrer">http://blog.brainlounge.de/memoryleaks/getting-started-with-spark-on-kubernetes/</a></li>
</ul>
<p>I have deployed my driver pod as is explained in the links above:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: spark
  name: my-notebook-deployment
  labels:
    app: my-notebook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-notebook
  template:
    metadata:
      labels:
        app: my-notebook
    spec:
      serviceAccountName: spark
      containers:
      - name: my-notebook
        image: pidocker-docker-registry.default.svc.cluster.local:5000/my-notebook:latest
        ports:
          - containerPort: 8888
        volumeMounts:
          - mountPath: /root/data
            name: my-notebook-pv
        workingDir: /root
        resources:
          limits:
            memory: 2Gi
      volumes:
        - name: my-notebook-pv
          persistentVolumeClaim:
            claimName: my-notebook-pvc
---
apiVersion: v1
kind: Service
metadata:
  namespace: spark
  name: my-notebook-deployment
spec:
  selector:
    app: my-notebook
  ports:
    - protocol: TCP
      port: 29413
  clusterIP: None
</code></pre>
<p>Then I can create the spark cluster using the following code:</p>
<pre class="lang-py prettyprint-override"><code>import os
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession
# Create Spark config for our Kubernetes based cluster manager
sparkConf = SparkConf()
sparkConf.setMaster(&quot;k8s://https://kubernetes.default.svc.cluster.local:443&quot;)
sparkConf.setAppName(&quot;spark&quot;)
sparkConf.set(&quot;spark.kubernetes.container.image&quot;, &quot;&lt;MYIMAGE&gt;&quot;)
sparkConf.set(&quot;spark.kubernetes.namespace&quot;, &quot;spark&quot;)
sparkConf.set(&quot;spark.executor.instances&quot;, &quot;7&quot;)
sparkConf.set(&quot;spark.executor.cores&quot;, &quot;2&quot;)
sparkConf.set(&quot;spark.driver.memory&quot;, &quot;512m&quot;)
sparkConf.set(&quot;spark.executor.memory&quot;, &quot;512m&quot;)
sparkConf.set(&quot;spark.kubernetes.pyspark.pythonVersion&quot;, &quot;3&quot;)
sparkConf.set(&quot;spark.kubernetes.authenticate.driver.serviceAccountName&quot;, &quot;spark&quot;)
sparkConf.set(&quot;spark.kubernetes.authenticate.serviceAccountName&quot;, &quot;spark&quot;)
sparkConf.set(&quot;spark.driver.port&quot;, &quot;29413&quot;)
sparkConf.set(&quot;spark.driver.host&quot;, &quot;my-notebook-deployment.spark.svc.cluster.local&quot;)
# Initialize our Spark cluster, this will actually
# generate the worker nodes.
spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()
sc = spark.sparkContext
</code></pre>
<p>It works.</p>
<p>How can I create an external pod that can execute a python script that lives in my my-notebook-deployment, I can do it in my terminal:</p>
<pre class="lang-sh prettyprint-override"><code>kubectl exec my-notebook-deployment-7669bb6fc-29stw python3 myscript.py
</code></pre>
<p>But I would want to be able to automate it executing this command inside another pod</p>

<p>In general you can spin up new pod with specified command running in it i.e.:</p>
<pre><code>kubectl run mypod --image=python3 --command -- &lt;cmd&gt; &lt;arg1&gt; ... &lt;argN&gt;
</code></pre>
<p>In your case you would need to provide the code of the myscript.py to the pod (i.e.: by mounting a ConfigMap with the script content) or build a new container image based on the python docker and adding the script to the image.</p>
