<p>I have a <a href="https://github.com/CoderPraBhu/coderprabhu-api/blob/master/k8s/coderprabhu-api-deployment.yaml" rel="nofollow noreferrer">Spring boot app</a> that talks to Mongo DB using spring-boot-starter-data-mongodb. I am using bootBuildImage to build the container. To make the app work under 180 MB, I tried to tweak as follows:   </p>

<p><strong>build.gradle</strong></p>

<pre><code>bootBuildImage {
    imageName = "gcr.io/kubegcp-256806/coderprabhu-api:${project.version}"
    environment = [
            "BPL_JVM_HEAD_ROOM" : "2",
            "BPL_JVM_LOADED_CLASS_COUNT" : "35",
            "BPL_JVM_THREAD_COUNT" : "10"
        ]
}
</code></pre>

<p><strong>deployment.yaml</strong></p>

<pre><code>        resources:
          requests:
            cpu: "0.02"
            memory: "64Mi"
          limits:
            cpu: "0.3"  
            memory: "180Mi" 
        env:
        - name: SPRING_PROFILES_ACTIVE
          value: "prod"    
        - name: BPL_JVM_HEAD_ROOM
          value: "2"  
        - name: BPL_JVM_LOADED_CLASS_COUNT
          value: "35"  
        - name: BPL_JVM_THREAD_COUNT
          value: "10"  
        - name: JAVA_OPTS
          value: &gt;-
                -XX:ReservedCodeCacheSize=80M
                -XX:MaxMetaspaceSize=60M
                -Xlog:gc
                -Xms110m
                -Xmx140m
                -XX:MaxRAM=180M
                -XX:+PrintFlagsFinal
</code></pre>

<p>Build pack that bootBuildImage uses has 240M for just ReservedCodeCache size. So tried to squeeze is down and spring boot with spring data mongo db seems to be ok with 80MB. I get OOMKilled if MaxMetaspaceSize is less than 60 MB. I only have 2 tomcat threads to serve here. </p>

<p><strong>Final GC Flags</strong></p>

<pre><code>kubectl logs coderprabhu-api-app-pod | grep command

      int ActiveProcessorCount  = 1            {product} {command line}
   size_t InitialHeapSize       = 115343360    {product} {command line}
   size_t MaxHeapSize           = 146800640    {product} {command line}
   size_t MaxMetaspaceSize      = 62914560     {product} {command line}
 uint64_t MaxRAM                = 188743680 {pd product} {command line}
     bool PrintFlagsFinal       = true         {product} {command line}
    uintx ReservedCodeCacheSize = 83886080  {pd product} {command line}
</code></pre>

<p>The container gets OOMKilled after some idle time and I see following logs. </p>

<pre><code>allocated memory is greater than 180M available for allocation: -XX:MaxDirectMemorySize=10M, -Xmx140M, -XX:MaxMetaspaceSize=60M, -XX:ReservedCodeCacheSize=80M, -Xss1M x 10 threads%                                                                                    
</code></pre>

<p>The node/express app on same server is happily serving complex functions at less than 30MB of total usage.</p>

<p>What is the smallest maximum memory a simple Spring Boot container can work with without getting OOMKilled?</p>

<p><strong>Out of Memory error cause:</strong> </p>

<p>In the deployment yaml in question, I had JVM arg <code>-XX:MaxRAM=180M</code> and also container limit <code>memory: "180Mi"</code>. This made JVM to think that it has access to all 180M and with no consideration of memory taken up by OS processes, causing out of memory errors. I increased container limit to <code>memory: "190Mi"</code> and reduced memory visible to JVM using <code>-XX:MaxRAM=150M</code>. 40MB buffer should be more than enough for occasional <code>kubectl exec -it</code>s for <code>bash</code> terminal and <code>top</code> command and other OS overheads. </p>

<p><strong>Further memory usage reduction:</strong></p>

<p>For this simple CRUD app the default Thread stack size of 1MB was too much and I was able to lower it to 256k. <code>-Xss256k</code> </p>

<p>From VisualVM, app uses max meta space of 41MB so, I set max meta space of 60 MB. <code>-XX:MaxMetaspaceSize=60M</code></p>

<p>Also Visual VM showed that after GC heap space comes down to 20 MB, so I set Max heap space to 40 MB to give some wiggle room. <code>-Xmx40m</code></p>

<p>Update on Xms on 6/11: Xms of 30MB was causing higher startup time and eventually heap memory was increasing to 32 MB anyways. So I have updated to <code>-Xms34m</code>. Startup now completes in 27 seconds. </p>

<p>The app has been running with no out of memory errors for past 40 hours.</p>

<p><strong>Kubernetes Stats:</strong></p>

<p><code>watch 'kubectl get pods|grep coderprabhu-api &amp; kubectl top pods|grep coderprabhu-api' 
coderprabhu-api-app-55b5774544-xgkxv   1/1     Running   0          20h
coderprabhu-api-app-55b5774544-xgkxv   2m           164Mi
</code></p>

<p><strong>GC Logs:</strong></p>

<p><code>Heap after GC invocations=182 (full 3): def new generation   total 12288K, used 130K 
  eden space 10944K,   0% used 
  from space 1344K,   9% used 
  to   space 1344K,   0% used 
 tenured generation   total 27328K, used 20475K 
   the space 27328K,  74% used 
 Metaspace       used 54241K, capacity 55807K, committed 55984K, reserved 102400K
  class space    used 6472K, capacity 7135K, committed 7168K, reserved 53248K
</code></p>

<p>I have been loading the app every once in a while with 500 requests and saw 90th percentile response of 115ms </p>

<p><code>ab -n 500 -c 2 https://api.coderprabhu.com/count</code></p>

<p>The pod has been using around 171 to 175MB of memory as seen on Google Cloud Dashboard. </p>

<p><strong>Google Cloud Usage</strong>
<a href="https://i.stack.imgur.com/quUPM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/quUPM.png" alt="Google cloud usage"></a></p>

<p>Following two articles from Dave Syer have been useful: </p>

<p><a href="https://spring.io/blog/2015/12/10/spring-boot-memory-performance" rel="nofollow noreferrer">Spring Boot Memory Performance</a></p>

<p><a href="https://spring.io/blog/2018/12/12/how-fast-is-spring" rel="nofollow noreferrer">How Fast is Spring?</a></p>

<p><strong>Current configuration:</strong>
<code>
      containers:
      - name: coderprabhu-api-app
        image: gcr.io/kubegcp-256806/coderprabhu-api:0.0.9-SNAPSHOT
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "0.02"
            memory: "64Mi"
          limits:
            cpu: "0.3" 
            memory: "190Mi" 
        env:
        - name: SPRING_PROFILES_ACTIVE
          value: "prod" 
        - name: BPL_JVM_HEAD_ROOM
          value: "2" 
        - name: BPL_JVM_LOADED_CLASS_COUNT
          value: "35" 
        - name: BPL_JVM_THREAD_COUNT
          value: "10" 
        - name: JAVA_OPTS
          value: &gt;-
                -XX:ReservedCodeCacheSize=40M
                -XX:MaxMetaspaceSize=60M
                -Xlog:gc*=debug
                -Xms34m
                -Xmx40m
                -Xss256k
                -XX:MaxRAM=150M
                -XX:+PrintFlagsFinal
</code></p>
