<p>I have a service running on Kubernetes processing files passed from another resource. Single file size can vary between 10MB - 1GB.</p>

<p>Recently I've been seeing the pod dead because of OOMKilled Error:</p>

<pre><code>State: Running
Started: Sun, 11 Nov 2018 07:28:46 +0000
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Fri, 09 Nov 2018 18:49:46 +0000
Finished: Sun, 11 Nov 2018 07:28:45 +0000
</code></pre>

<p>I mitigate the issue by bumping the resource (Memory) limit on the pod. But I am concerning whenever there is a traffic or file size spike, we will run into this OOMKilled issue again. But if I set the memory limit too high, I am concerning it will cause trouble on the host of this pod. </p>

<p>I read through the best practices given by Kubernetes: <a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#best-practices" rel="noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#best-practices</a>. But I am not sure by adding --eviction-hard and --system-reserved=memory could resolve the issue. </p>

<p>Has anyone had experience with a similar issue before? </p>

<p>Any help would be appreciated.</p>

<p>More than a Kubernetes/Container runtime issue this is more memory management in your application and this will vary depending on what language runtime or if something like the JVM is running your application.</p>

<p>You generally want to set an upper limit on the memory usage in the application, for example, a maximum heap space in your JVM, then leave a little headroom for garbage collection and overruns. </p>

<p>Another example is the Go runtime and looks like <a href="https://github.com/golang/go/issues/16843" rel="noreferrer">they have talked</a> about memory management but with no solution as of this writing. For these cases, it might be good to manually set the <a href="https://unix.stackexchange.com/a/44987/54556"><code>ulimit</code></a> the virtual memory for the specific process of your application. (If you have a leak you will see other types of errors) or using <a href="https://unix.stackexchange.com/a/44988/54556">timeout</a></p>

<p>There's also <a href="https://unix.stackexchange.com/a/125024/54556">manual cgroup management</a> but then again that's exactly with docker and Kubernetes are supposed to do.</p>

<p><a href="https://jaxenter.com/nobody-puts-java-container-139373.html" rel="noreferrer">This is a good article</a> with some insights about managing a JVM in containers.</p>
