<p>I am trying to get mongo to be scheduled to a given node in my cluster (qatar).</p>
<p>I see the following error message in the pod description:</p>
<pre><code>  Warning  FailedScheduling  58m   default-scheduler  0/7 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 6 node(s) didn't find available persistent volumes to bind.
</code></pre>
<p>Mongo relies on the following 2 claims:</p>
<pre><code>[dsargrad@malta cfg]$ kubectl get pvc
NAME                                    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
data-volume-learning-center-mongodb-0   Pending                                      local-storage   3m57s
logs-volume-learning-center-mongodb-0   Pending                                      local-storage   3m57s

[dsargrad@malta cfg]$ kubectl describe pvc data-volume-learning-center-mongodb-0
Name:          data-volume-learning-center-mongodb-0
Namespace:     default
StorageClass:  local-storage
Status:        Pending
Volume:
Labels:        app=learning-center-mongodb-svc
Annotations:   &lt;none&gt;
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       learning-center-mongodb-0
Events:
  Type    Reason                Age                   From                         Message
  ----    ------                ----                  ----                         -------
  Normal  WaitForFirstConsumer  4m45s                 persistentvolume-controller  waiting for first consumer to be created before binding
  Normal  WaitForPodScheduled   12s (x19 over 4m42s)  persistentvolume-controller  waiting for pod learning-center-mongodb-0 to be scheduled
</code></pre>
<p>My two PV's that I want to be bound are as follows:</p>
<pre><code>[dsargrad@malta cfg]$ kubectl get pv
NAME            CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                           STORAGECLASS    REASON   AGE
mongo-data-pv   1Gi        RWO            Retain           Available   default/data-volume-learning-center-mongodb-0   local-storage            8m47s
mongo-logs-pv   1Gi        RWO            Retain           Available   default/logs-volume-learning-center-mongodb-0   local-storage            15m
</code></pre>
<p>These use &quot;local&quot; storage.. on the qatar.corp.sensis.com node.</p>
<pre><code>[dsargrad@malta cfg]$ kubectl get nodes
NAME                    STATUS   ROLES                  AGE   VERSION
benin.corp.sensis.com   Ready    &lt;none&gt;                 45h   v1.20.5
chad.corp.sensis.com    Ready    &lt;none&gt;                 45h   v1.20.5
malta.corp.sensis.com   Ready    control-plane,master   45h   v1.20.5
qatar.corp.sensis.com   Ready    &lt;none&gt;                 45h   v1.20.5
spain.corp.sensis.com   Ready    &lt;none&gt;                 45h   v1.20.5
togo.corp.sensis.com    Ready    &lt;none&gt;                 45h   v1.20.5
tonga.corp.sensis.com   Ready    &lt;none&gt;                 45h   v1.20.5
</code></pre>
<p>My mongo pod wont schedule</p>
<pre><code>[dsargrad@malta cfg]$ kubectl describe pod learning-center-mongodb-0
Name:           learning-center-mongodb-0
Namespace:      default
Priority:       0
Node:           &lt;none&gt;
Labels:         app=learning-center-mongodb-svc
                controller-revision-hash=learning-center-mongodb-784678577f
                statefulset.kubernetes.io/pod-name=learning-center-mongodb-0
Annotations:    &lt;none&gt;
Status:         Pending
IP:
IPs:            &lt;none&gt;
Controlled By:  StatefulSet/learning-center-mongodb
Init Containers:
  mongod-posthook:
    Image:      quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.2
    Port:       &lt;none&gt;
    Host Port:  &lt;none&gt;
    Command:
      cp
      version-upgrade-hook
      /hooks/version-upgrade
    Environment:  &lt;none&gt;
    Mounts:
      /hooks from hooks (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from mongodb-kubernetes-operator-token-ldwsr (ro)
  mongodb-agent-readinessprobe:
    Image:      quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.1
    Port:       &lt;none&gt;
    Host Port:  &lt;none&gt;
    Command:
      cp
      /probes/readinessprobe
      /opt/scripts/readinessprobe
    Environment:  &lt;none&gt;
    Mounts:
      /opt/scripts from agent-scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from mongodb-kubernetes-operator-token-ldwsr (ro)
Containers:
  mongod:
    Image:      registry.hub.docker.com/library/mongo:4.2.6
    Port:       &lt;none&gt;
    Host Port:  &lt;none&gt;
    Command:
      /bin/sh
      -c

      #run post-start hook to handle version changes
      /hooks/version-upgrade

      # wait for config and keyfile to be created by the agent
       while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;


      # start mongod with this configuration
      exec mongod -f /data/automation-mongod.conf;

    Limits:
      cpu:     1
      memory:  500M
    Requests:
      cpu:     500m
      memory:  400M
    Environment:
      AGENT_STATUS_FILEPATH:  /healthstatus/agent-health-status.json
    Mounts:
      /data from data-volume (rw)
      /healthstatus from healthstatus (rw)
      /hooks from hooks (rw)
      /var/lib/mongodb-mms-automation/authentication from learning-center-mongodb-keyfile (rw)
      /var/log/mongodb-mms-automation from logs-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from mongodb-kubernetes-operator-token-ldwsr (ro)
  mongodb-agent:
    Image:      quay.io/mongodb/mongodb-agent:10.27.0.6772-1
    Port:       &lt;none&gt;
    Host Port:  &lt;none&gt;
    Command:
      /bin/bash
      -c
      current_uid=$(id -u)
      echo $current_uid
      declare -r current_uid
      if ! grep -q &quot;${current_uid}&quot; /etc/passwd ; then
      sed -e &quot;s/^mongodb:/builder:/&quot; /etc/passwd &gt; /tmp/passwd
      echo &quot;mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash&quot; &gt;&gt; /tmp/passwd
      cat /tmp/passwd
      export NSS_WRAPPER_PASSWD=/tmp/passwd
      export LD_PRELOAD=libnss_wrapper.so
      export NSS_WRAPPER_GROUP=/etc/group
      fi
      agent/mongodb-agent -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -useLocalMongoDbTools
    Limits:
      cpu:     1
      memory:  500M
    Requests:
      cpu:      500m
      memory:   400M
    Readiness:  exec [/opt/scripts/readinessprobe] delay=5s timeout=1s period=10s #success=1 #failure=60
    Environment:
      AGENT_STATUS_FILEPATH:  /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
      AUTOMATION_CONFIG_MAP:  learning-center-mongodb-config
      HEADLESS_AGENT:         true
      POD_NAMESPACE:          default (v1:metadata.namespace)
    Mounts:
      /data from data-volume (rw)
      /opt/scripts from agent-scripts (rw)
      /var/lib/automation/config from automation-config (ro)
      /var/lib/mongodb-mms-automation/authentication from learning-center-mongodb-keyfile (rw)
      /var/log/mongodb-mms-automation from logs-volume (rw)
      /var/log/mongodb-mms-automation/healthstatus from healthstatus (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from mongodb-kubernetes-operator-token-ldwsr (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  logs-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  logs-volume-learning-center-mongodb-0
    ReadOnly:   false
  data-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-volume-learning-center-mongodb-0
    ReadOnly:   false
  agent-scripts:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  &lt;unset&gt;
  automation-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  learning-center-mongodb-config
    Optional:    false
  healthstatus:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  &lt;unset&gt;
  hooks:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  &lt;unset&gt;
  learning-center-mongodb-keyfile:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  &lt;unset&gt;
  mongodb-kubernetes-operator-token-ldwsr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  mongodb-kubernetes-operator-token-ldwsr
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  7m19s  default-scheduler  0/7 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 6 node(s) didn't find available persistent volumes to bind.
  Warning  FailedScheduling  7m19s  default-scheduler  0/7 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 6 node(s) didn't find available persistent volumes to bind.
</code></pre>
<p>I use the claimRef in the creation of the PV.</p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongo-logs-pv
  labels:
    app: learning-center-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  claimRef:
    namespace: default
    name: logs-volume-learning-center-mongodb-0
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /home/storage/mongo/logs
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - qatar.corp.sensis.com
</code></pre>
<p>My local-storage class:</p>
<pre><code>[dsargrad@malta cfg]$ kubectl get storageclass
NAME                      PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-storage (default)   kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  4h22m
</code></pre>
<p>Here is a description of the data PV</p>
<pre><code>[dsargrad@malta cfg]$ kubectl describe pv mongo-data-pv
Name:              mongo-data-pv
Labels:            app=learning-center-mongodb-svc
Annotations:       &lt;none&gt;
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-storage
Status:            Available
Claim:             default/data-volume-learning-center-mongodb-0
Reclaim Policy:    Retain
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          1Gi
Node Affinity:
  Required Terms:
    Term 0:        kubernetes.io/hostname in [qatar.corp.sensis.com]
Message:
Source:
    Type:  LocalVolume (a persistent volume backed by local storage on a node)
    Path:  /home/storage/mongo/data
Events:    &lt;none&gt;
</code></pre>
<p>and the logs PV</p>
<pre><code>[dsargrad@malta cfg]$ kubectl describe pv mongo-logs-pv
Name:              mongo-logs-pv
Labels:            app=learning-center-mongodb-svc
Annotations:       &lt;none&gt;
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-storage
Status:            Available
Claim:             default/logs-volume-learning-center-mongodb-0
Reclaim Policy:    Retain
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          1Gi
Node Affinity:
  Required Terms:
    Term 0:        kubernetes.io/hostname in [qatar.corp.sensis.com]
Message:
Source:
    Type:  LocalVolume (a persistent volume backed by local storage on a node)
    Path:  /home/storage/mongo/logs
Events:    &lt;none&gt;
</code></pre>
<p>On the node qatar.corp.sensis.com I have the folders referenced in the PV
<a href="https://i.stack.imgur.com/DsVie.png" rel="nofollow noreferrer">Screenshot of directory with permissions</a></p>
<p>Why wont the pod schedule to qatar.corp.sensis.com and the PVCs bind to the PVs?</p>

<p>I made the boneheaded assumption that if the PVC had claimed a size I'd see that in the output of the <em><strong>describe</strong></em> command. I had to get the yaml for the PVC spec to see that it requested more than the PV had allocated</p>
<p>I've now successfully bound</p>
<pre><code>[dsargrad@malta cfg]$ kubectl apply -f *logs* --namespace default
persistentvolume/mongo-logs-pv configured
[dsargrad@malta cfg]$ kubectl apply -f *data* --namespace default
persistentvolume/mongo-data-pv configured
[dsargrad@malta cfg]$ kubectl get pvc
NAME                                    STATUS   VOLUME          CAPACITY   ACCESS MODES   STORAGECLASS    AGE
data-volume-learning-center-mongodb-0   Bound    mongo-data-pv   10Gi       RWO            local-storage   98m
logs-volume-learning-center-mongodb-0   Bound    mongo-logs-pv   10Gi       RWO            local-storage   98m
</code></pre>
<p>I found the detail I needed by looking carefully at the PVC spec.
Interestingly it was <a href="https://www.youtube.com/watch?v=0CFb26BNeTQ&amp;t=301s" rel="nofollow noreferrer">this</a> video on youtube that clued me to the answer. See time starting at about 6:50.</p>
<p>In the following note the storage size is &quot;2G&quot;.</p>
<pre><code>[dsargrad@malta cfg]$ kubectl get pvc logs-volume-learning-center-mongodb-0 -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: &quot;2021-03-31T15:55:40Z&quot;
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: learning-center-mongodb-svc
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:app: {}
      f:spec:
        f:accessModes: {}
        f:resources:
          f:requests:
            .: {}
            f:storage: {}
        f:volumeMode: {}
      f:status:
        f:phase: {}
    manager: kube-controller-manager
    operation: Update
    time: &quot;2021-03-31T15:55:40Z&quot;
  name: logs-volume-learning-center-mongodb-0
  namespace: default
  resourceVersion: &quot;302313&quot;
  uid: 09ef80fe-a45e-45e4-b515-9746b9265476
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2G
  storageClassName: local-storage
  volumeMode: Filesystem
status:
  phase: Pending
</code></pre>
