<p>I recently pushed a new container image to one of my GKE deployments and noticed that API latency went up and requests started returning 502's.</p>
<p>Looking at the logs I found that the container started crashing because of OOM:</p>
<pre><code>Memory cgroup out of memory: Killed process 2774370 (main) total-vm:1801348kB, anon-rss:1043688kB, file-rss:12884kB, shmem-rss:0kB, UID:0 pgtables:2236kB oom_score_adj:980
</code></pre>
<p>Looking at the memory usage graph it didn't look like the pods were using more than 50MB of memory combined. My original resource requests were:</p>
<pre><code>...
spec:
...
  template:
...
    spec:
...
      containers:
      - name: api-server
...
        resources:
          # You must specify requests for CPU to autoscale
          # based on CPU utilization
          requests:
            cpu: &quot;150m&quot;
            memory: &quot;80Mi&quot;
          limits:
            cpu: &quot;1&quot;
            memory: &quot;1024Mi&quot;
      - name: cloud-sql-proxy
        # It is recommended to use the latest version of the Cloud SQL proxy
        # Make sure to update on a regular schedule!
        image: gcr.io/cloudsql-docker/gce-proxy:1.17
        resources:
          # You must specify requests for CPU to autoscale
          # based on CPU utilization
          requests:
            cpu: &quot;100m&quot;
...
</code></pre>
<p>Then I tried bumping the request for API server to 1GB but it did not help. In the end, what helped was reverting the container image to the previous version:</p>
<p><a href="https://i.stack.imgur.com/pcbOr.png" rel="noreferrer"><img src="https://i.stack.imgur.com/pcbOr.png" alt="enter image description here" /></a></p>
<p>Looking through the changes in the golang binary there are no obvious memory leaks. When I run it locally it uses at most 80MB of memory, even under load from the same requests as in production.</p>
<p>And the above graph which I got from the GKE console also shows the pod using far less than the 1GB memory limit.</p>
<p>So my question is: What could cause GKE to kill my process for OOM when both GKE monitoring and running it locally only uses 80MB out of the 1GB limit?</p>
<p>=== EDIT ===</p>
<p><a href="https://i.stack.imgur.com/m0aKD.png" rel="noreferrer"><img src="https://i.stack.imgur.com/m0aKD.png" alt="enter image description here" /></a></p>
<p>Adding another graph of the same outage. This time splitting the two containers in the pod. If I understand correctly, the metric here is <a href="https://cloud.google.com/monitoring/api/metrics_kubernetes?_ga=2.93050717.-1793855474.1618969918" rel="noreferrer">non-evictable container/memory/used_bytes</a>:</p>
<pre><code>container/memory/used_bytes GA
Memory usage
GAUGE, INT64, By
k8s_container   Memory usage in bytes. Sampled every 60 seconds.
memory_type: Either `evictable` or `non-evictable`. Evictable memory is memory that can be easily reclaimed by the kernel, while non-evictable memory cannot.
</code></pre>
<h2>Edit Apr 26 2021</h2>
<p>I tried updating the resources field in the deployment yaml to 1GB RAM requested and 1GB RAM limit like suggested by Paul and Ryan:</p>
<pre><code>        resources:
          # You must specify requests for CPU to autoscale
          # based on CPU utilization
          requests:
            cpu: &quot;150m&quot;
            memory: &quot;1024Mi&quot;
          limits:
            cpu: &quot;1&quot;
            memory: &quot;1024Mi&quot;
</code></pre>
<p>Unfortunately it had the same result after updating with <code>kubectl apply -f api_server_deployment.yaml</code>:</p>
<pre><code>{
 insertId: &quot;yyq7u3g2sy7f00&quot;  
 jsonPayload: {
  apiVersion: &quot;v1&quot;   
  eventTime: null   
  involvedObject: {
   kind: &quot;Node&quot;    
   name: &quot;gke-api-us-central-1-e2-highcpu-4-nod-dfe5c3a6-c0jy&quot;    
   uid: &quot;gke-api-us-central-1-e2-highcpu-4-nod-dfe5c3a6-c0jy&quot;    
  }
  kind: &quot;Event&quot;   
  message: &quot;Memory cgroup out of memory: Killed process 1707107 (main) total-vm:1801412kB, anon-rss:1043284kB, file-rss:9732kB, shmem-rss:0kB, UID:0 pgtables:2224kB oom_score_adj:741&quot;   
  metadata: {
   creationTimestamp: &quot;2021-04-26T23:13:13Z&quot;    
   managedFields: [
    0: {
     apiVersion: &quot;v1&quot;      
     fieldsType: &quot;FieldsV1&quot;      
     fieldsV1: {
      f:count: {
      }
      f:firstTimestamp: {
      }
      f:involvedObject: {
       f:kind: {
       }
       f:name: {
       }
       f:uid: {
       }
      }
      f:lastTimestamp: {
      }
      f:message: {
      }
      f:reason: {
      }
      f:source: {
       f:component: {
       }
       f:host: {
       }
      }
      f:type: {
      }
     }
     manager: &quot;node-problem-detector&quot;      
     operation: &quot;Update&quot;      
     time: &quot;2021-04-26T23:13:13Z&quot;      
    }
   ]
   name: &quot;gke-api-us-central-1-e2-highcpu-4-nod-dfe5c3a6-c0jy.16798b61e3b76ec7&quot;    
   namespace: &quot;default&quot;    
   resourceVersion: &quot;156359&quot;    
   selfLink: &quot;/api/v1/namespaces/default/events/gke-api-us-central-1-e2-highcpu-4-nod-dfe5c3a6-c0jy.16798b61e3b76ec7&quot;    
   uid: &quot;da2ad319-3f86-4ec7-8467-e7523c9eff1c&quot;    
  }
  reason: &quot;OOMKilling&quot;   
  reportingComponent: &quot;&quot;   
  reportingInstance: &quot;&quot;   
  source: {
   component: &quot;kernel-monitor&quot;    
   host: &quot;gke-api-us-central-1-e2-highcpu-4-nod-dfe5c3a6-c0jy&quot;    
  }
  type: &quot;Warning&quot;   
 }
 logName: &quot;projects/questions-279902/logs/events&quot;  
 receiveTimestamp: &quot;2021-04-26T23:13:16.918764734Z&quot;  
 resource: {
  labels: {
   cluster_name: &quot;api-us-central-1&quot;    
   location: &quot;us-central1-a&quot;    
   node_name: &quot;gke-api-us-central-1-e2-highcpu-4-nod-dfe5c3a6-c0jy&quot;    
   project_id: &quot;questions-279902&quot;    
  }
  type: &quot;k8s_node&quot;   
 }
 severity: &quot;WARNING&quot;  
 timestamp: &quot;2021-04-26T23:13:13Z&quot;  
}
</code></pre>
<p>Kubernetes seems to have almost immediately killed the container for using 1GB of memory. But again, the metrics show that container using only 2MB of memory:</p>
<p><a href="https://i.stack.imgur.com/OJ09D.png" rel="noreferrer"><img src="https://i.stack.imgur.com/OJ09D.png" alt="enter image description here" /></a></p>
<p>And again I am stumped because even under load this binary does not use more than 80MB when I run it locally.</p>
<p>I also tried running <code>go tool pprof &lt;url&gt;/debug/pprof/heap</code>. It showed several different values as Kubernetes kept thrashing the container. But none higher than ~20MB and not memory usage out of the ordinary</p>
<h2>Edit 04/27</h2>
<p>I tried setting request=limit for both containers in the pod:</p>
<pre><code> requests:
   cpu: &quot;1&quot;
   memory: &quot;1024Mi&quot;
 limits:
   cpu: &quot;1&quot;
   memory: &quot;1024Mi&quot;
...
requests:
  cpu: &quot;100m&quot;
  memory: &quot;200Mi&quot;
limits:
  cpu: &quot;100m&quot;
  memory: &quot;200Mi&quot;
</code></pre>
<p>But it didn't work either:</p>
<pre><code>Memory cgroup out of memory: Killed process 2662217 (main) total-vm:1800900kB, anon-rss:1042888kB, file-rss:10384kB, shmem-rss:0kB, UID:0 pgtables:2224kB oom_score_adj:-998
</code></pre>
<p>And the memory metrics still show usage in the single digit MBs.</p>
<h2>Update 04/30</h2>
<p>I pinpointed the change that seemed to cause this issue by painstakingly checking out my latest commits one by one.</p>
<p>In the offending commit I had a couple of lines like</p>
<pre><code>type Pic struct {
        image.Image
        Proto *pb.Image
}
...

pic.Image = picture.Resize(pic, sz.Height, sz.Width)
...
</code></pre>
<p>Where <code>picture.Resize</code> eventually calls <a href="https://pkg.go.dev/github.com/nfnt/resize#Resize" rel="noreferrer"><code>resize.Resize</code></a>.
I changed it to:</p>
<pre><code>type Pic struct {
        Img   image.Image
        Proto *pb.Image
 }
...
pic.Img = picture.Resize(pic.Img, sz.Height, sz.Width)
</code></pre>
<p>This solves my immediate problem and the container runs fine now. But it does not answer my original question:</p>
<ol>
<li>Why did these lines cause GKE to OOM my container?</li>
<li>And why did the GKE memory metrics show that everything was fine?</li>
</ol>

<p>I guess it was caused by <a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/" rel="nofollow noreferrer">Pod QoS class</a></p>
<p>When the system is overcommitted, the QoS classes determine which pod gets killed first so the freed resources can be given to higher priority pods.</p>
<p>In your case, the QoS of your pod would be <a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#qos-classes" rel="nofollow noreferrer"><code>Burstable</code></a></p>
<p>Each running process has an OutOfMemory(OOM) score. The system selects the process to kill by comparing OOM score of all the running processes. When memory needs to be freed, the process with the highest score gets killed. For details of how the <code>score</code> is calculated please refer to <a href="https://serverfault.com/questions/571319/how-is-kernel-oom-score-calculated">How is kernel oom score calculated?</a>.</p>
<p><strong>Which pod will be killed first if both in the <code>Burstable</code> class?</strong></p>
<p>For short, the system will kill the one using more of its requested memory than the other in percentage-wise.</p>
<pre><code>Pod A

used: 90Mi
requests: 100Mi
limits: 200Mi
</code></pre>
<pre><code>Pod B

used: 150Mi
requests: 200Mi
limits: 400Mi
</code></pre>
<p><code>Pod A</code> will get killed before <code>Pod B</code> because it uses 90% of its requested memory while <code>Pod B</code> use only 75% of its requested memory.</p>
