title: Kubernetes - AutoScaling doesn't increase the number of nodes when pod limit has reached
body: <p>Update: found that instances has a limit to number of <a href="https://stackoverflow.com/a/57971006/8445363">pods</a>.</p>
  <p>Does that mean that auto-scaling group won't increase the node count unless the CPU or memory resources hit that limit?</p>
  <p>I'm running <code>t3.medium</code> instance and looking at the nodes describe command output it hasn't used 100% of resources.  But I get the error <code>0/1 nodes are available: 1 Too many pods.</code>.</p>
  <p>Another thing is I have auto scaling configured for cluster node group, and the max size is 2 but second node hasn't spawn yet which means there is still resources right?</p>
  <pre><code>  scaling_config {
      desired_size  = 1
      max_size      = 2
      min_size      = 1
    }
  </code></pre>
  <p>Can someone explain why it can't run new pod and what's blocking it?</p>
  <p><code>kubectl get po -A</code>:</p>
  <pre><code>NAMESPACE        NAME                                                            READY   STATUS    RESTARTS   AGE
  default          alertmanager-prometheus-prometheus-oper-alertmanager-0          2/2     Running   0          3h34m
  default          mongo-express-5b8b8989c8-b7qq8                                  1/1     Running   0          3h33m
  default          mongodb-0                                                       1/1     Running   0          3h37m
  default          mongodb-1                                                       1/1     Running   0          3h36m
  default          mongodb-arbiter-0                                               1/1     Running   0          3h36m
  default          mongodb-exporter-prometheus-mongodb-exporter-786999d67c-c586f   1/1     Running   0          123m
  default          nginx-ingress-controller-868b5465b4-8klbr                       1/1     Running   0          3h45m
  default          nginx-ingress-default-backend-7c868597f4-n85s8                  1/1     Running   0          3h45m
  default          prometheus-grafana-67596ff846-nkssr                             2/2     Running   0          3h35m
  default          prometheus-kube-state-metrics-c65b87574-g6pt7                   1/1     Running   0          3h35m
  default          prometheus-prometheus-node-exporter-kl5b2                       1/1     Running   0          3h35m
  default          prometheus-prometheus-oper-operator-5ff8fbd5fb-fzgvf            2/2     Running   0          3h35m
  default          prometheus-prometheus-prometheus-oper-prometheus-0              3/3     Running   1          3h34m
  kube-system      aws-node-7t29d                                                  1/1     Running   0          5h58m
  kube-system      coredns-6b4cdc67b4-7mnjh                                        1/1     Running   0          6h2m
  kube-system      coredns-6b4cdc67b4-7px74                                        1/1     Running   0          6h2m
  kube-system      kube-proxy-dl2dr                                                1/1     Running   0          5h58m
  test-db-dev   db01-0                                                          0/1     Pending   0          11m
  
  
  `kubectl describe node`:
  
      Name:               ip-10-0-0-250.ap-south-1.compute.internal
      Roles:              &lt;none&gt;
      Labels:             beta.kubernetes.io/arch=amd64
                          beta.kubernetes.io/instance-type=t3.medium
                          beta.kubernetes.io/os=linux
                          eks.amazonaws.com/capacityType=ON_DEMAND
                          eks.amazonaws.com/nodegroup=test_nodes-dev
                          eks.amazonaws.com/nodegroup-image=ami-0ad87e82a7f1e7f4b
                          failure-domain.beta.kubernetes.io/region=ap-south-1
                          failure-domain.beta.kubernetes.io/zone=ap-south-1b
                          kubernetes.io/arch=amd64
                          kubernetes.io/hostname=ip-10-0-0-250.ap-south-1.compute.internal
                          kubernetes.io/os=linux
                          node.kubernetes.io/instance-type=t3.medium
                          topology.kubernetes.io/region=ap-south-1
                          topology.kubernetes.io/zone=ap-south-1b
      Annotations:        node.alpha.kubernetes.io/ttl: 0
                          volumes.kubernetes.io/controller-managed-attach-detach: true
      CreationTimestamp:  Wed, 09 Dec 2020 03:43:22 -0500
      Taints:             &lt;none&gt;
      Unschedulable:      false
      Lease:
        HolderIdentity:  ip-10-0-0-250.ap-south-1.compute.internal
        AcquireTime:     &lt;unset&gt;
        RenewTime:       Wed, 09 Dec 2020 09:39:11 -0500
      Conditions:
        Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
        ----             ------  -----------------                 ------------------                ------                       -------
        MemoryPressure   False   Wed, 09 Dec 2020 09:39:04 -0500   Wed, 09 Dec 2020 03:43:22 -0500   KubeletHasSufficientMemory   kubelet has sufficient memory available
        DiskPressure     False   Wed, 09 Dec 2020 09:39:04 -0500   Wed, 09 Dec 2020 03:43:22 -0500   KubeletHasNoDiskPressure     kubelet has no disk pressure
        PIDPressure      False   Wed, 09 Dec 2020 09:39:04 -0500   Wed, 09 Dec 2020 03:43:22 -0500   KubeletHasSufficientPID      kubelet has sufficient PID available
        Ready            True    Wed, 09 Dec 2020 09:39:04 -0500   Wed, 09 Dec 2020 03:43:42 -0500   KubeletReady                 kubelet is posting ready status
      Addresses:
        InternalIP:   10.0.0.250
        Hostname:     ip-10-0-0-250.ap-south-1.compute.internal
        InternalDNS:  ip-10-0-0-250.ap-south-1.compute.internal
      Capacity:
        attachable-volumes-aws-ebs:  25
        cpu:                         2
        ephemeral-storage:           20959212Ki
        hugepages-1Gi:               0
        hugepages-2Mi:               0
        memory:                      3977908Ki
        pods:                        17
      Allocatable:
        attachable-volumes-aws-ebs:  25
        cpu:                         1930m
        ephemeral-storage:           18242267924
        hugepages-1Gi:               0
        hugepages-2Mi:               0
        memory:                      3422900Ki
        pods:                        17
      System Info:
        Machine ID:                 ec2474d1824a7ec75cfb9c1704c92d34
        System UUID:                EC2474D1-824A-7EC7-5CFB-9C1704C92D34
        Boot ID:                    3ddcd32a-8b7a-4a21-b41b-37a51686ca67
        Kernel Version:             4.14.203-156.332.amzn2.x86_64
        OS Image:                   Amazon Linux 2
        Operating System:           linux
        Architecture:               amd64
        Container Runtime Version:  docker://19.3.6
        Kubelet Version:            v1.18.9-eks-d1db3c
        Kube-Proxy Version:         v1.18.9-eks-d1db3c
      ProviderID:                   aws:///ap-south-1b/i-047e645215dc3c7e1
      Non-terminated Pods:          (17 in total)
        Namespace                   Name                                                             CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
        ---------                   ----                                                             ------------  ----------  ---------------  -------------  ---
        default                     alertmanager-prometheus-prometheus-oper-alertmanager-0           100m (5%)     100m (5%)   225Mi (6%)       25Mi (0%)      3h32m
        default                     mongo-express-5b8b8989c8-b7qq8                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h31m
        default                     mongodb-0                                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h34m
        default                     mongodb-1                                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h34m
        default                     mongodb-arbiter-0                                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h34m
        default                     mongodb-exporter-prometheus-mongodb-exporter-786999d67c-c586f    0 (0%)        0 (0%)      0 (0%)           0 (0%)         121m
        default                     nginx-ingress-controller-868b5465b4-8klbr                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h43m
        default                     nginx-ingress-default-backend-7c868597f4-n85s8                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h43m
        default                     prometheus-grafana-67596ff846-nkssr                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h32m
        default                     prometheus-kube-state-metrics-c65b87574-g6pt7                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h32m
        default                     prometheus-prometheus-node-exporter-kl5b2                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h32m
        default                     prometheus-prometheus-oper-operator-5ff8fbd5fb-fzgvf             0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h32m
        default                     prometheus-prometheus-prometheus-oper-prometheus-0               200m (10%)    200m (10%)  50Mi (1%)        50Mi (1%)      3h32m
        kube-system                 aws-node-7t29d                                                   10m (0%)      0 (0%)      0 (0%)           0 (0%)         5h55m
        kube-system                 coredns-6b4cdc67b4-7mnjh                                         100m (5%)     0 (0%)      70Mi (2%)        170Mi (5%)     6h
        kube-system                 coredns-6b4cdc67b4-7px74                                         100m (5%)     0 (0%)      70Mi (2%)        170Mi (5%)     6h
        kube-system                 kube-proxy-dl2dr                                                 100m (5%)     0 (0%)      0 (0%)           0 (0%)         5h55m
      Allocated resources:
        (Total limits may be over 100 percent, i.e., overcommitted.)
        Resource                    Requests     Limits
        --------                    --------     ------
        cpu                         610m (31%)   300m (15%)
        memory                      415Mi (12%)  415Mi (12%)
        ephemeral-storage           0 (0%)       0 (0%)
        hugepages-1Gi               0 (0%)       0 (0%)
        hugepages-2Mi               0 (0%)       0 (0%)
        attachable-volumes-aws-ebs  0            0
      Events:                       &lt;none&gt;
  </code></pre>