<p>I'm trying to deploy a Prometheus nodeexporter Daemonset in my AWS EKS K8s cluster.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: prometheus
    chart: prometheus-11.12.1
    component: node-exporter
    heritage: Helm
    release: prometheus
  name: prometheus-node-exporter
  namespace: operations-tools-test
spec:
  selector:
    matchLabels:
      app: prometheus
      component: node-exporter
      release: prometheus
  template:
    metadata:
      labels:
        app: prometheus
        chart: prometheus-11.12.1
        component: node-exporter
        heritage: Helm
        release: prometheus
    spec:
      containers:
      - args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --web.listen-address=:9100
        image: prom/node-exporter:v1.0.1
        imagePullPolicy: IfNotPresent
        name: prometheus-node-exporter
        ports:
        - containerPort: 9100
          hostPort: 9100
          name: metrics
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /host/proc
          name: proc
          readOnly: true
        - mountPath: /host/sys
          name: sys
          readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: true
      hostPID: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: prometheus-node-exporter
      serviceAccountName: prometheus-node-exporter
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /proc
          type: &quot;&quot;
        name: proc
      - hostPath:
          path: /sys
          type: &quot;&quot;
        name: sys
</code></pre>
<p>After deploying however, its not getting deployed on one node.</p>
<p>pod.yml file for that file looks like this:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: eks.privileged
  generateName: prometheus-node-exporter-
  labels:
    app: prometheus
    chart: prometheus-11.12.1
    component: node-exporter
    heritage: Helm
    pod-template-generation: &quot;1&quot;
    release: prometheus
  name: prometheus-node-exporter-xxxxx
  namespace: operations-tools-test
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: prometheus-node-exporter
  resourceVersion: &quot;51496903&quot;
  selfLink: /api/v1/namespaces/namespace-x/pods/prometheus-node-exporter-xxxxx
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - ip-xxx-xx-xxx-xxx.ec2.internal
  containers:
  - args:
    - --path.procfs=/host/proc
    - --path.sysfs=/host/sys
    - --web.listen-address=:9100
    image: prom/node-exporter:v1.0.1
    imagePullPolicy: IfNotPresent
    name: prometheus-node-exporter
    ports:
    - containerPort: 9100
      hostPort: 9100
      name: metrics
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /host/proc
      name: proc
      readOnly: true
    - mountPath: /host/sys
      name: sys
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: prometheus-node-exporter-token-xxxx
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostNetwork: true
  hostPID: true
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: prometheus-node-exporter
  serviceAccountName: prometheus-node-exporter
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/pid-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/network-unavailable
    operator: Exists
  volumes:
  - hostPath:
      path: /proc
      type: &quot;&quot;
    name: proc
  - hostPath:
      path: /sys
      type: &quot;&quot;
    name: sys
  - name: prometheus-node-exporter-token-xxxxx
    secret:
      defaultMode: 420
      secretName: prometheus-node-exporter-token-xxxxx
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: &quot;2020-11-06T23:56:47Z&quot;
    message: '0/4 nodes are available: 2 node(s) didn''t have free ports for the requested
      pod ports, 3 Insufficient pods, 3 node(s) didn''t match node selector.'
    reason: Unschedulable
    status: &quot;False&quot;
    type: PodScheduled
  phase: Pending
  qosClass: BestEffort
</code></pre>
<p>As seen above POD nodeAffinity looks up metadata.name which matches exactly what I have as a label in my node.</p>
<p>But when I run the below command,</p>
<pre class="lang-sh prettyprint-override"><code> kubectl describe  po prometheus-node-exporter-xxxxx
</code></pre>
<p>I get in the events:</p>
<pre><code>Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  60m                   default-scheduler  0/4 nodes are available: 1 Insufficient pods, 3 node(s) didn't match node selector.
  Warning  FailedScheduling  4m46s (x37 over 58m)  default-scheduler  0/4 nodes are available: 2 node(s) didn't have free ports for the requested pod ports, 3 Insufficient pods, 3 node(s) didn't match node selector.
</code></pre>
<p>I have also checked Cloud-watch logs for Scheduler and I don't see any logs for my failed pod.</p>
<p>The Node has ample resources left</p>
<pre><code>Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests    Limits
  --------                    --------    ------
  cpu                         520m (26%)  210m (10%)
  memory                      386Mi (4%)  486Mi (6%)
</code></pre>
<p>I don't see a reason why it should not schedule a pod.
Can anyone help me with this?</p>
<p>TIA</p>

Shubham Shrivastav
<p>As posted in the comments:</p>
<blockquote>
<p>Please add to the question the steps that you followed (editing any values in the Helm chart etc). Also please <strong>check if the nodes are not over the limit of pods that can be scheduled on it</strong>. Here you can find the link for more reference: <a href="https://stackoverflow.com/questions/52898067/aks-reporting-insufficient-pods">LINK</a>.</p>
</blockquote>
<blockquote>
<p>no processes occupying 9100 on the given node. <strong>@DawidKruk The POD limit was reached. Thanks!</strong> I expected them to give me some error regarding that rather than vague node selector property not matching</p>
</blockquote>
<hr />
<p>Not really sure why the following messages were displayed:</p>
<ul>
<li>node(s) didn't have free ports for the requested pod ports</li>
<li>node(s) didn't match node selector</li>
</ul>
<p><strong>The issue that <code>Pods</code> couldn't be scheduled on the nodes (<code>Pending</code> state) was connected with the <code>Insufficient pods</code> message in the <code>$ kubectl get events</code> command.</strong></p>
<p>Above message is displayed when the nodes reached their maximum capacity of pods (example: <code>node1</code> can schedule maximum of <code>30</code> pods).</p>
<hr />
<p>More on the <code>Insufficient Pods</code> can be found in this github issue comment:</p>
<blockquote>
<p>That's true. That's because the CNI implementation on EKS. Max pods number is limited by the network interfaces attached to instance multiplied by the number of ips per ENI - which varies depending on the size of instance. It's apparent for small instances, this number can be quite a low number.</p>
<p><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI" rel="nofollow noreferrer">Docs.aws.amazon.com: AWSEC2: User Guide: Using ENI: Available IP per ENI</a></p>
<p>-- <em><a href="https://github.com/kubernetes/autoscaler/issues/1576#issuecomment-454100551" rel="nofollow noreferrer">Github.com: Kubernetes: Autoscaler: Issue 1576: Comment 454100551</a></em></p>
</blockquote>
<hr />
<p>Additional resources:</p>
<ul>
<li><em><a href="https://stackoverflow.com/questions/57970896/pod-limit-on-node-aws-eks">Stackoverflow.com: Questions: Pod limit on node AWS EKS</a></em></li>
</ul>
