<p>I need some help regarding this OOM status of pods 137. I am kinda stuck here for 3 days now. I built a docker image of a flask application. I run the docker image, it was running fine with a memory usage of 2.7 GB.
I uploaded it to GKE with the following specification.
<a href="https://i.stack.imgur.com/o5dGZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/o5dGZ.png" alt="Cluster" /></a></p>
<p>Workloads show &quot;Does not have minimum availability&quot;</p>
<p><a href="https://i.stack.imgur.com/860ox.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/860ox.png" alt="Workload" /></a></p>
<p>Checking that pod &quot;news-category-68797b888c-fmpnc&quot; shows CrashLoopBackError</p>
<p>Error &quot;back-off 5m0s restarting failed container=news-category-pbz8s pod=news-category-68797b888c-fmpnc_default(d5b0fff8-3e35-4f14-8926-343e99195543): CrashLoopBackOff&quot;</p>
<p>Checking the YAML file shows OOM killed 137.</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: &quot;2022-02-17T16:07:36Z&quot;
  generateName: news-category-68797b888c-
  labels:
    app: news-category
    pod-template-hash: 68797b888c
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName: {}
        f:labels:
          .: {}
          f:app: {}
          f:pod-template-hash: {}
        f:ownerReferences:
          .: {}
          k:{&quot;uid&quot;:&quot;8d99448a-04f6-4651-a652-b1cc6d0ae4fc&quot;}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:containers:
          k:{&quot;name&quot;:&quot;news-category-pbz8s&quot;}:
            .: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:resources: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
        f:dnsPolicy: {}
        f:enableServiceLinks: {}
        f:restartPolicy: {}
        f:schedulerName: {}
        f:securityContext: {}
        f:terminationGracePeriodSeconds: {}
    manager: kube-controller-manager
    operation: Update
    time: &quot;2022-02-17T16:07:36Z&quot;
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{&quot;type&quot;:&quot;ContainersReady&quot;}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
          k:{&quot;type&quot;:&quot;Initialized&quot;}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{&quot;type&quot;:&quot;Ready&quot;}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
        f:containerStatuses: {}
        f:hostIP: {}
        f:phase: {}
        f:podIP: {}
        f:podIPs:
          .: {}
          k:{&quot;ip&quot;:&quot;10.16.3.4&quot;}:
            .: {}
            f:ip: {}
        f:startTime: {}
    manager: kubelet
    operation: Update
    time: &quot;2022-02-17T16:55:18Z&quot;
  name: news-category-68797b888c-fmpnc
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: news-category-68797b888c
    uid: 8d99448a-04f6-4651-a652-b1cc6d0ae4fc
  resourceVersion: &quot;25100&quot;
  uid: d5b0fff8-3e35-4f14-8926-343e99195543
spec:
  containers:
  - image: gcr.io/projectiris-327708/news_category:noConsoleDebug
    imagePullPolicy: IfNotPresent
    name: news-category-pbz8s
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-z2lbp
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: gke-news-category-cluste-default-pool-42e1e905-ftzb
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-z2lbp
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: &quot;2022-02-17T16:07:37Z&quot;
    status: &quot;True&quot;
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: &quot;2022-02-17T16:55:18Z&quot;
    message: 'containers with unready status: [news-category-pbz8s]'
    reason: ContainersNotReady
    status: &quot;False&quot;
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: &quot;2022-02-17T16:55:18Z&quot;
    message: 'containers with unready status: [news-category-pbz8s]'
    reason: ContainersNotReady
    status: &quot;False&quot;
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: &quot;2022-02-17T16:07:36Z&quot;
    status: &quot;True&quot;
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://a582af0248a330b7d4087916752bd941949387ed708f00b3aac6f91a6ef75e63
    image: gcr.io/projectiris-327708/news_category:noConsoleDebug
    imageID: gcr.io/projectiris-327708/news_category@sha256:c4b3385bd80eff2a0c0ec0df18c6a28948881e2a90dd1c642ec6960b63dd017a
    lastState:
      terminated:
        containerID: containerd://a582af0248a330b7d4087916752bd941949387ed708f00b3aac6f91a6ef75e63
        exitCode: 137
        finishedAt: &quot;2022-02-17T16:55:17Z&quot;
        reason: OOMKilled
        startedAt: &quot;2022-02-17T16:54:48Z&quot;
    name: news-category-pbz8s
    ready: false
    restartCount: 13
    started: false
    state:
      waiting:
        message: back-off 5m0s restarting failed container=news-category-pbz8s pod=news-category-68797b888c-fmpnc_default(d5b0fff8-3e35-4f14-8926-343e99195543)
        reason: CrashLoopBackOff
  hostIP: 10.160.0.42
  phase: Running
  podIP: 10.16.3.4
  podIPs:
  - ip: 10.16.3.4
  qosClass: BestEffort
  startTime: &quot;2022-02-17T16:07:37Z&quot;

</code></pre>
<p>My question is what to do and how to do to solve this. I tried to add resources to the YAML file in spec-</p>
<pre><code>resources:
   limits:
       memory: 32Gi
   requests:
       memory: 16Gi
</code></pre>
<p>It also shows errors. How do I increase the memory of pods? And also if I increase memory it shows &quot;Pod Unscheduled&quot;.</p>
<p>Someone plz give me an insight into clusters, nodes, and pods and how to solve this. Thank you.</p>

Pritam Sinha
<p>The message <code>back-off restarting failed container</code> appears when you are facing a temporary resource overload, as a result of an activity spike. And the <code>OOMKilled code 137</code> means that a container or pod was terminated because they used more memory than the one allowed. OOM stands for <code>“Out Of Memory”</code>.</p>
<p>So based on this information from your GKE configuration  (Total of memory 16Gi ), I suggest review the total of memory limited configured in your GKE, you can confirm this issue with the following command:  <code>kubectl describe pod [name]</code></p>
<p>You will need to determine why Kubernetes decided to terminate the pod with the OOMKilled error, and adjust memory requests and limit values, here is an example to increase your memory and how to set limits to the <a href="https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/" rel="nofollow noreferrer">memory</a> because looks like this is the main problem</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: memory-demo
  namespace: mem-example
spec:
  containers:
  - name: memory-demo-ctr
    image: polinux/stress
    resources:
      limits:
        memory: &quot;128Gi&quot;
      requests:
        memory: &quot;64Gi&quot;
    command: [&quot;stress&quot;]
    args: [&quot;--vm&quot;, &quot;1&quot;, &quot;--vm-bytes&quot;, &quot;64Gi&quot;, &quot;--vm-hang&quot;, &quot;1&quot;]
</code></pre>
<p>To review all the GKE metrics, you go to the GCP console, then go to the Monitoring dashboard and select GKE. In this Monitoring dashboard, you will find the statistics related to memory and CPU.</p>
<p>Also, it is important to review if the containers have enough resources to run your applications. You can follow this <a href="https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-resource-requests-and-limits" rel="nofollow noreferrer">link</a> to check the Kubernetes best practices.</p>
<p>GKE also has an interesting feature, which is autoscaling. It is very useful because it automatically resizes your GKE cluster based on the demands of your workloads. Please follow this <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler" rel="nofollow noreferrer">guide</a> to learn how to do it.</p>
