<p>I am creating a spark cluster on Kubernetes and following the official documentation:
<a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes</a></p>
<p>As mentioned in the above link <code>hostPath</code> is one of the volume mount supported type for Spark cluster on Kubernetes.</p>
<p>I have created a cluster using <code>kubeadm init</code> command and using <code>spark-submit</code> tool to submit a spark job which creates driver and executors pods on demand.</p>
<p>I need to read some large files from this Spark application and <code>hostPath</code> seemed to be the only realistic option to mount this files on the cluster at this Proof of concept stage.</p>
<p>How can I create a <code>hostPath</code> volume on a Kubernetes cluster without an existing deployment yaml file? Since I am using <code>spark-submit</code> command, I don't have a yaml config file.</p>
<p>Trying to create a yaml file with kind <code>hostPath</code> is not a valid option.</p>
<p>Is there any other way using <code>kubectl</code> to create a <code>hostPath</code> volume or any other option with yaml config file?</p>

adesai
<p>As @david-maze mentioned, <code>hostPath</code> is rarely a good option for pods since we shouldn't always schedule them into the same node.</p>
<p>However, if it's for test purposes on a local Kubernetes cluster. <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/" rel="nofollow noreferrer">Here is a doc</a> describing how to create a hostPath PV and how to claim it using a PVC to mount into you spark pods.</p>
<p>Basically, here is the PV yaml</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1

kind: PersistentVolume

metadata:

  name: task-pv-volume

  labels:

    type: local

spec:

  storageClassName: manual

  capacity:

    storage: 10Gi

  accessModes:

    - ReadWriteOnce

  hostPath:

    path: &quot;/mnt/data&quot;

</code></pre>
<p>Then</p>
<pre><code>kubectl apply -f /path/to/pv-volume.yaml
</code></pre>
<p>Here is the PVC yaml</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: task-pv-claim

spec:

  storageClassName: manual

  accessModes:

    - ReadWriteOnce

  resources:

    requests:

      storage: 10Gi

</code></pre>
<p>Then</p>
<pre><code>kubectl apply -f /path/to/pv-claim.yaml
</code></pre>
<p>Then you use the pvc in the deployment resource you have or using a pod manifest</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1

kind: Pod

metadata:

  name: spark-worker

spec:

  volumes:

    - name: task-pv-storage

      persistentVolumeClaim:

        claimName: task-pv-claim

  containers:

    - name: spark-worker

      image: spark-hadoop:3.2.0

      command: [&quot;/spark-worker&quot;]

      ports:

        - containerPort: 8081

      volumeMounts:

        - mountPath: &quot;/usr/share/nginx/html&quot;

          name: task-pv-storage

</code></pre>
