<p>In my Kubernetes cluster I want to define a StatefulSet using a local persistence volume on each node. My Kubernetes cluster has worker nodes.</p>
<ul>
<li>worker-node-1</li>
<li>worker-node-2</li>
<li>worker-node-3</li>
</ul>
<p>My <code>StatefulSet</code> looks something like this:</p>
<pre><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: myset
spec:
  replicas: 3
  ...
  template:
    spec:
     ....
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - myset
              topologyKey: kubernetes.io/hostname
      containers:
     ....
        volumeMounts:
        - name: datadir
          mountPath: /data
      volumes:
      - name: datadir
        persistentVolumeClaim:
          claimName: datadir
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes:
        - &quot;ReadWriteOnce&quot;
      storageClassName: &quot;local-storage&quot;
      resources:
        requests:
          storage: 10Gi
</code></pre>
<p>I want to achieve, that on each POD, running on a separate node, a local data volume is used.</p>
<p>I defined a <code>StorageClass</code> object:</p>
<pre><code>kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
</code></pre>
<p>and the following <code>PersistentVolume</code>:</p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /var/lib/my-data/
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker-node-1
</code></pre>
<p>But of course, this did not work as I have defined a <code>nodeAffinity</code> with only the hostname for my first worker-node-1. As a result I can see only one PV. The PVC and the POD on the corresponding node starts as expected. But on the other two nodes I have no PVs. How can I define, that a local <code>PersistenceVolume</code> is created for each worker-node?</p>
<p>I also tried to define a nodeAffinity with 3 values:</p>
<pre><code>  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker-node-1
          - worker-node-2
          - worker-node-3
</code></pre>
<p>But this also did not work.</p>

Ralph
<blockquote>
<p><em>I fear that the <code>PersitenceVolume</code> I define is the problem. This object
will create exactly one <code>PV</code> and so only one of my PODs finds the
corresponding <code>PV</code> and can be scheduled.</em></p>
</blockquote>
<p>Yes, you're right. By creating <code>PersistentVolume</code> object, you create exactly <strong>one</strong> <code>PersistentVolume</code>. No more, no less. If you define 3 separate <code>PVs</code> that can be available on each of your 3 nodes, you shouldn't experience any problem.</p>
<p>If you have, let's say, 3 worker nodes, you need to create 3 separate <code>PersistentVolumes</code>, each one with different <code>NodeAffinity</code>. <strong>You don't need to define any <code>NodeAffinity</code> in your <code>StatefulSet</code> as it is already handled on <code>PersistentVolume</code> level and should be defined only there.</strong></p>
<p>As you can read in the <a href="https://kubernetes.io/docs/concepts/storage/volumes/#local" rel="nofollow noreferrer">local volume</a> documentation:</p>
<blockquote>
<p>Compared to <code>hostPath</code> volumes, <code>local</code> volumes are used in a durable and
portable manner without manually scheduling pods to nodes. The system
is aware of the volume's node constraints by looking at the node
affinity on the PersistentVolume.</p>
</blockquote>
<p><strong>Remember: PVC -&gt; PV mapping is always 1:1. You cannot bind 1 PVC to 3 different PVs or the other way.</strong></p>
<blockquote>
<p><em>So my only solution is to switch form local PV to hostPath volumes which is working fine.</em></p>
</blockquote>
<p>Yes, it can be done with <code>hostpath</code> but I wouldn't say it is the only and the best solution. Local volumes have several advantages over hostpath volumes and it is worth considering choosing them. But as I mentioned above, in your use case you need to create 3 separate <code>PVs</code> manually. You already created one <code>PV</code> so it shouldn't be a big deal to create another two. This is the way to go.</p>
<blockquote>
<p><em>I want to achieve, that on each POD, running on a separate node, a
local data volume is used.</em></p>
</blockquote>
<p>It can be achieved with local volumes but in such case instead of using a single PVC in your <code>StatefulSet</code> definition as in the below fragment from your configuration:</p>
<pre><code>  volumes:
  - name: datadir
    persistentVolumeClaim:
      claimName: datadir
</code></pre>
<p>you need to use only <code>volumeClaimTemplates</code> as in <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#components" rel="nofollow noreferrer">this</a> example, which may look as follows:</p>
<pre><code>  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ &quot;ReadWriteOnce&quot; ]
      storageClassName: &quot;my-storage-class&quot;
      resources:
        requests:
          storage: 1Gi
</code></pre>
<p>As you can see, the <code>PVCs</code> won't &quot;look&quot; for a <code>PV</code> with any particular name so you can name them as you wish. They will &quot;look&quot; for a <code>PV</code> belonging to a particular <code>StorageClass</code> and in this particular case supporting <code>&quot;ReadWriteOnce&quot;</code> <code>accessMode</code>.</p>
<p>The scheduler will attempt to find the adequate node, on which your stateful pod can be scheduled. If another pod was already scheduled, let's say, on <code>worker-1</code> and the only <code>PV</code> belonging to our <code>local-storage</code> storage class isn't available any more, the scheduler will try to find another node that meets storage requirements. So again: no need for node affinity/ pod antiaffinity rules in your <code>StatefulSet</code> definition.</p>
<blockquote>
<p><em>But I need some mechanism that a PV is created for each node and
assigned with the PODs created by the StatefulSet. But this did not
work - I always have only one PV.</em></p>
</blockquote>
<p>In order to facilitate the management of volumes and automate the whole process to certain extent, take a look at <a href="https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner" rel="nofollow noreferrer">Local Persistence Volume Static Provisioner</a>. As its name may already suggest, it doesn't support dynamic provisioning (as we have e.g. on various cloud platforms), which means you are still responsible for creating the underlying storage but the whole volume lifecycle can be handled automatically.</p>
<p>To make this whole theoretical explanation somewhat more practical, I'm adding below a working example, which you can quickly test for yourself. Make sure <code>/var/tmp/test</code> directory is created on every nodes or adjust the below examples to your needs:</p>
<p><code>StatefulSet</code> components (slightly modified example from <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#components" rel="nofollow noreferrer">here</a>):</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  serviceName: &quot;nginx&quot;
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ &quot;ReadWriteOnce&quot; ]
      storageClassName: &quot;local-storage&quot;
      resources:
        requests:
          storage: 1Gi
</code></pre>
<p><code>StorageClass</code> definition:</p>
<pre><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
</code></pre>
<p>And finally a <code>PV</code>. You need to make 3 versions of the below yaml manifest by setting different names e.g. <code>example-pv-1</code>,<code>example-pv-2</code> and <code>example-pv-3</code> and node names.</p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-pv-1 ### 👈 change it
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /var/tmp/test ### 👈 you can adjust shared directory on the node 
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker-node-1 ### 👈 change this value by setting your node name
</code></pre>
<p>So 3 different <code>PVs</code> for 3 worker nodes.</p>
