<p>I have 2 pods, one that is writing files to a persistent volume and the other one supposedly reads those files to make some calculations.</p>
<p>The first pod writes the files successfully and when I display the content of the persistent volume using <code>print(os.listdir(persistent_volume_path))</code> I get all the expected files. However, the same command on the second pod shows an empty directory. (The mountPath directory <code>/data</code> is created but empty.)</p>
<p>This is the TFJob yaml file:</p>
<pre><code>apiVersion: kubeflow.org/v1
kind: TFJob
metadata:
  name: pod1
  namespace: my-namespace
spec:
  cleanPodPolicy: None
  tfReplicaSpecs:
    Worker:
      replicas: 1
      restartPolicy: Never
      template:
        spec:
          containers:
            - name: tensorflow
              image: my-image:latest
              imagePullPolicy: Always
              command:
                - &quot;python&quot;
                - &quot;./program1.py&quot;
                - &quot;--data_path=./dataset.csv&quot;
                - &quot;--persistent_volume_path=/data&quot;
              volumeMounts:
                - mountPath: &quot;/data&quot;
                  name: my-pv
          volumes:
            - name: my-pv
              persistentVolumeClaim:
                claimName: my-pvc
</code></pre>
<p>(respectively pod2 and program2.py for the second pod)</p>
<p>And this is the volume configuration:</p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
  namespace: my-namespace
  labels:
    type: local
    app: tfjob
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
</code></pre>
<hr />
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
  namespace: my-namespace
  labels:
    type: local
    app: tfjob
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: &quot;/data&quot;
</code></pre>
<p>Does anyone have any idea where's the problem exactly and how to fix it?</p>

<p>When two pods should access a shared Persistent Volume with <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes" rel="nofollow noreferrer">access mode</a> <code>ReadWriteOnce</code>, concurrently - then the two pods must be running on the <strong>same node</strong> since the volume can only be mounted on a single node at a time with this access mode.</p>
<p>To achieve this, some form of <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/" rel="nofollow noreferrer">Pod Affinity</a> must be applied, such that they are scheduled to the same node.</p>
