<p>I deployed a k8s cluster on AWS EKS fargate. And deployed a elasticsearch container to the pod. The pod is stuck on <code>ContainerCreating</code> state and <code>describe pod</code> shows below error:</p>
<pre><code>$ kubectl describe pod es-0
Name:                 es-0
Namespace:            default
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 fargate-ip-10-0-1-207.ap-southeast-2.compute.internal/10.0.1.207
Start Time:           Fri, 28 May 2021 16:39:07 +1000
Labels:               controller-revision-hash=es-86f54d94fb
                      eks.amazonaws.com/fargate-profile=elk_profile
                      name=es
                      statefulset.kubernetes.io/pod-name=es-0
Annotations:          CapacityProvisioned: 1vCPU 2GB
                      Logging: LoggingDisabled: LOGGING_CONFIGMAP_NOT_FOUND
                      kubernetes.io/psp: eks.privileged
Status:               Pending
IP:
IPs:                  &lt;none&gt;
Controlled By:        StatefulSet/es
Containers:
  es:
    Container ID:
    Image:          elasticsearch:7.10.1
    Image ID:
    Ports:          9200/TCP, 9300/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  8
    Requests:
      cpu:        1
      memory:     4
    Environment:  &lt;none&gt;
    Mounts:
      /usr/share/elasticsearch/config/elasticsearch.yml from es-config (rw,path=&quot;elasticsearch.yml&quot;)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6qql4 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  es-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      es-config
    Optional:  false
  default-token-6qql4:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6qql4
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                   From     Message
  ----     ------                  ----                  ----     -------
  Warning  FailedCreatePodSandBox  75s (x4252 over 16h)  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused &quot;process_linux.go:319: getting the final child's pid from pipe caused \&quot;read init-p: connection reset by peer\&quot;&quot;: unknown
</code></pre>
<p>How do I know what the issue is and how to fix it? I have tried to restart the <code>Statefulset</code> but it didn't restart. It seems the pod stucked.</p>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: es-config
data:
  elasticsearch.yml: |
    cluster.name: my-elastic-cluster
    network.host: &quot;0.0.0.0&quot;
    bootstrap.memory_lock: false
    discovery.zen.ping.unicast.hosts: elasticsearch-cluster
    discovery.zen.minimum_master_nodes: 1
    discovery.type: single-node
  ES_JAVA_OPTS: -Xms2g -Xmx4g
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es
  namespace: default
spec:
  serviceName: es-entrypoint
  replicas: 1
  selector:
    matchLabels:
      name: es
  template:
    metadata:
      labels:
        name: es
    spec:
      volumes:
        - name: es-config
          configMap:
            name: es-config
            items:
              - key: elasticsearch.yml
                path: elasticsearch.yml
        # - name: persistent-storage
        #   persistentVolumeClaim:
        #     claimName: efs-es-claim
      securityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsGroup: 1000
      containers:
        - name: es
          image: elasticsearch:7.10.1
          resources:
            limits:
              cpu: 2
              memory: 8
            requests:
              cpu: 1
              memory: 4
          ports:
            - name: http
              containerPort: 9200
            - containerPort: 9300
              name: inter-node
          volumeMounts:
            - name: es-config
              mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
              subPath: elasticsearch.yml
            # - name: persistent-storage
            #   mountPath: /usr/share/elasticsearch/data
---
apiVersion: v1
kind: Service
metadata:
  name: es-entrypoint
spec:
  selector:
    name: es
  ports:
    - port: 9200
      targetPort: 9200
      protocol: TCP
  type: NodePort
</code></pre>

Joey Yi Zhao
<p>In My case, it is the policy issue of eks cni.
If run :
<code>kubectl logs -n kube-system -l k8s-app=aws-node</code></p>
<pre><code>Installed /host/opt/cni/bin/egress-v4-cni
time=&quot;2023-05-07T03:09:29Z&quot; level=info msg=&quot;Starting IPAM daemon... &quot;
time=&quot;2023-05-07T03:09:29Z&quot; level=info msg=&quot;Checking for IPAM connectivity... &quot;
time=&quot;2023-05-07T03:09:30Z&quot; level=info msg=&quot;Copying config file... &quot;
time=&quot;2023-05-07T03:09:30Z&quot; level=info msg=&quot;Successfully copied CNI plugin binary and config file.&quot;
time=&quot;2023-05-07T03:09:30Z&quot; level=error msg=&quot;Failed to wait for IPAM daemon to complete&quot; error=&quot;exit status 1&quot;
Installed /host/opt/cni/bin/aws-cni
Installed /host/opt/cni/bin/egress-v4-cni
time=&quot;2023-05-07T03:09:35Z&quot; level=info msg=&quot;Starting IPAM daemon... &quot;
time=&quot;2023-05-07T03:09:35Z&quot; level=info msg=&quot;Checking for IPAM connectivity... &quot;
time=&quot;2023-05-07T03:09:36Z&quot; level=info msg=&quot;Copying config file... &quot;
time=&quot;2023-05-07T03:09:36Z&quot; level=info msg=&quot;Successfully copied CNI plugin binary and config file.&quot;
time=&quot;2023-05-07T03:09:36Z&quot; level=error msg=&quot;Failed to wait for IPAM daemon to complete&quot; error=&quot;exit status 1&quot;
</code></pre>
<p>Once I attach AmazonEKS_CNI_Policy to my worker node role. It works:</p>
<pre><code>Installed /host/opt/cni/bin/aws-cni
Installed /host/opt/cni/bin/egress-v4-cni
time=&quot;2023-05-07T03:11:55Z&quot; level=info msg=&quot;Starting IPAM daemon... &quot;
time=&quot;2023-05-07T03:11:55Z&quot; level=info msg=&quot;Checking for IPAM connectivity... &quot;
time=&quot;2023-05-07T03:11:56Z&quot; level=info msg=&quot;Copying config file... &quot;
time=&quot;2023-05-07T03:11:56Z&quot; level=info msg=&quot;Successfully copied CNI plugin binary and config file.&quot;
Installed /host/opt/cni/bin/aws-cni
Installed /host/opt/cni/bin/egress-v4-cni
time=&quot;2023-05-07T03:11:55Z&quot; level=info msg=&quot;Starting IPAM daemon... &quot;
time=&quot;2023-05-07T03:11:55Z&quot; level=info msg=&quot;Checking for IPAM connectivity... &quot;
time=&quot;2023-05-07T03:11:56Z&quot; level=info msg=&quot;Copying config file... &quot;
time=&quot;2023-05-07T03:11:56Z&quot; level=info msg=&quot;Successfully copied CNI plugin binary and config file.&quot;
</code></pre>
