<p>Good afternoon</p>
<p>I really need some help getting a group of sentinels up so that they can monitor and perform elections for my redis pods, which are running without issue. At the bottom of this message I have included the sentinel config, which spells out the volumes. The first sentinel, sentinel0, sits at Pending, while the rest of the redis instances are READY 1/1, for all three.</p>
<p>But they don't get scheduled. When I attempt to apply the sentinel statefulset, I get the following schedule error. The sentinel statefulset config is at the bottom of this post</p>
<blockquote>
<p>Warning  FailedScheduling  5s    default-scheduler  0/4 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) didn't find available persistent volumes to bind.
Warning  FailedScheduling  4s    default-scheduler  0/4 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 3 node(s) didn't find available persistent volumes to bind.</p>
</blockquote>
<p>About my kubernetes setup:</p>
<p>I am running a four-node baremetal kubernetes cluster; one master node and three worker nodes respectively.</p>
<p>For storage, I am using a 'local-storage' StorageClass shared across the nodes. Currently I am using a single persistent volume configuration file which defines three volumes across three nodes. This seems to be working out for the redis statefulset, but not sentinel. (sentiel config at bottom)</p>
<p>See below config of persistent volume (all three pv-volume-node-0, 1, 2 all are bound)</p>
<pre><code>kind: PersistentVolume
apiVersion: v1
metadata:
  name: ag1-pv-volume-node-0
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: &quot;/var/opt/mssql&quot;
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node-0
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: ag1-pv-volume-node-1
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: &quot;/var/opt/mssql&quot;
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node-1
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: ag1-pv-volume-node-2
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: &quot;/var/opt/mssql&quot;
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node-2
</code></pre>
<p>Note: the path &quot;/var/opt/mssql&quot; is the stateful directory data pt for the redis cluster. It's a misnomer and in no way reflects a sql database (I just used this directory from a walkthrough), and it works.</p>
<p>Presently all three redis pods are successfully deployed with a functioning statefulset, see below for the redis config (all working)</p>
<pre><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  serviceName: redis
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
      - name: config
        image: redis:6.0-alpine
        command: [ &quot;sh&quot;, &quot;-c&quot; ]
        args:
          - |
            cp /tmp/redis/redis.conf /etc/redis/redis.conf
            
            echo &quot;finding master...&quot;
            MASTER_FDQN=`hostname  -f | sed -e 's/redis-[0-9]\./redis-0./'`
            if [ &quot;$(redis-cli -h sentinel -p 5000 ping)&quot; != &quot;PONG&quot; ]; then
              echo &quot;master not found, defaulting to redis-0&quot;

              if [ &quot;$(hostname)&quot; == &quot;redis-0&quot; ]; then
                echo &quot;this is redis-0, not updating config...&quot;
              else
                echo &quot;updating redis.conf...&quot;
                echo &quot;slaveof $MASTER_FDQN 6379&quot; &gt;&gt; /etc/redis/redis.conf
              fi
            else
              echo &quot;sentinel found, finding master&quot;
              MASTER=&quot;$(redis-cli -h sentinel -p 5000 sentinel get-master-addr-by-name mymaster | grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}')&quot;
              echo &quot;master found : $MASTER, updating redis.conf&quot;
              echo &quot;slaveof $MASTER 6379&quot; &gt;&gt; /etc/redis/redis.conf
            fi
        volumeMounts:
        - name: redis-config
          mountPath: /etc/redis/
        - name: config
          mountPath: /tmp/redis/
      containers:
      - name: redis
        image: redis:6.0-alpine
        command: [&quot;redis-server&quot;]
        args: [&quot;/etc/redis/redis.conf&quot;]
        ports:
        - containerPort: 6379
          name: redis
        volumeMounts:
        - name: data
          mountPath: /var/opt/mssql
        - name: redis-config
          mountPath: /etc/redis/
      volumes:
      - name: redis-config
        emptyDir: {}
      - name: config
        configMap:
          name: redis-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ &quot;ReadWriteOnce&quot; ]
      storageClassName: &quot;local-storage&quot;
      resources:
        requests:
          storage: 50Mi
---
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  clusterIP: None
  ports:
  - port: 6379
    targetPort: 6379
    name: redis
  selector:
    app: redis
</code></pre>
<p>The real issue I'm having, I believe spawns from how I've configured the sentinel statefulset. The pods won't schedule and its printed reason is it isn't finding persistent volumes to bind from.</p>
<p>SENTINEL STATEFULSET CONFIG, problem here, can't figure out how to set it up right with the volumes I made.</p>
<pre><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sentinel
spec:
  serviceName: sentinel
  replicas: 3
  selector:
    matchLabels:
      app: sentinel
  template:
    metadata:
      labels:
        app: sentinel
    spec:
      initContainers:
      - name: config
        image: redis:6.0-alpine
        command: [ &quot;sh&quot;, &quot;-c&quot; ]
        args:
          - |
            REDIS_PASSWORD=a-very-complex-password-here
            nodes=redis-0.redis.redis.svc.cluster.local,redis-1.redis.redis.svc.cluster.local,redis-2.redis.redis.svc.cluster.local

            for i in ${nodes//,/ }
            do
                echo &quot;finding master at $i&quot;
                MASTER=$(redis-cli --no-auth-warning --raw -h $i -a $REDIS_PASSWORD info replication | awk '{print $1}' | grep master_host: | cut -d &quot;:&quot; -f2)
                if [ &quot;$MASTER&quot; == &quot;&quot; ]; then
                    echo &quot;no master found&quot;
                    MASTER=
                else
                    echo &quot;found $MASTER&quot;
                    break
                fi
            done
            echo &quot;sentinel monitor mymaster $MASTER 6379 2&quot; &gt;&gt; /tmp/master

            echo &quot;port 5000
            $(cat /tmp/master)
            sentinel down-after-milliseconds mymaster 5000
            sentinel failover-timeout mymaster 60000
            sentinel parallel-syncs mymaster 1
            sentinel auth-pass mymaster $REDIS_PASSWORD
            &quot; &gt; /etc/redis/sentinel.conf
            cat /etc/redis/sentinel.conf
        volumeMounts:
        - name: redis-config
          mountPath: /etc/redis/
      containers:
      - name: sentinel
        image: redis:6.0-alpine
        command: [&quot;redis-sentinel&quot;]
        args: [&quot;/etc/redis/sentinel.conf&quot;]
        ports:
        - containerPort: 5000
          name: sentinel
        volumeMounts:
        - name: redis-config
          mountPath: /etc/redis/
        - name: data
          mountPath: /var/opt/mssql
      volumes:
      - name: redis-config
        emptyDir: {}
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ &quot;ReadWriteOnce&quot; ]
      storageClassName: &quot;local-storage&quot;
      resources:
        requests:
          storage: 50Mi
---
apiVersion: v1
kind: Service
metadata:
  name: sentinel
spec:
  clusterIP: None
  ports:
  - port: 5000
    targetPort: 5000
    name: sentinel
  selector:
    app: sentinel
</code></pre>
<p>This is my first post here. I am a big fan of stackoverflow!</p>

007chungking
<p>You may try to create three PVs using this template:</p>
<pre><code>kind: PersistentVolume
apiVersion: v1
metadata:
  name: ag1-pv-volume-node-0
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  claimRef:
    namespace: default
    name: data-redis-0
  local:
    path: &quot;/var/opt/mssql&quot;
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node-0
</code></pre>
<p>Important part here is <code>claimRef</code> field which ties PV with PVC with StatefulSet.
It should be of special format.</p>
<p>Read more here: <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/preexisting-pd#using_a_preexisting_disk_in_a_statefulset" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/preexisting-pd#using_a_preexisting_disk_in_a_statefulset</a></p>
