<p>I logged into the Linux bastion host where <code>kubectl</code> is installed for connecting to the Kubernetes cluster.</p>
<p>On the Bastion host when I run any kubectl command like the one below:</p>
<pre><code>kubectl get pods
</code></pre>
<p>I get the error below:</p>
<pre><code>fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x192cb47, 0x16)
        /usr/local/go/src/runtime/panic.go:617 +0x72 fp=0x7fff534ae560 sp=0x7fff534ae530 pc=0x42d482
runtime.sysMap(0xc000000000, 0x4000000, 0x2d178b8)
        /usr/local/go/src/runtime/mem_linux.go:170 +0xc7 fp=0x7fff534ae5a0 sp=0x7fff534ae560 pc=0x417d07
runtime.(*mheap).sysAlloc(0x2cfef80, 0x2000, 0x2cfef90, 0x1)
        /usr/local/go/src/runtime/malloc.go:633 +0x1cd fp=0x7fff534ae648 sp=0x7fff534ae5a0 pc=0x40ae0d
runtime.(*mheap).grow(0x2cfef80, 0x1, 0x0)
        /usr/local/go/src/runtime/mheap.go:1222 +0x42 fp=0x7fff534ae6a0 sp=0x7fff534ae648 pc=0x425022
runtime.(*mheap).allocSpanLocked(0x2cfef80, 0x1, 0x2d178c8, 0x0)
        /usr/local/go/src/runtime/mheap.go:1150 +0x37f fp=0x7fff534ae6d8 sp=0x7fff534ae6a0 pc=0x424f0f
runtime.(*mheap).alloc_m(0x2cfef80, 0x1, 0x2a, 0x0)
        /usr/local/go/src/runtime/mheap.go:977 +0xc2 fp=0x7fff534ae728 sp=0x7fff534ae6d8 pc=0x424562
runtime.(*mheap).alloc.func1()
        /usr/local/go/src/runtime/mheap.go:1048 +0x4c fp=0x7fff534ae760 sp=0x7fff534ae728 pc=0x456a4c
runtime.(*mheap).alloc(0x2cfef80, 0x1, 0x1002a, 0x0)
        /usr/local/go/src/runtime/mheap.go:1047 +0x8a fp=0x7fff534ae7b0 sp=0x7fff534ae760 pc=0x42483a
runtime.(*mcentral).grow(0x2cffd80, 0x0)
        /usr/local/go/src/runtime/mcentral.go:256 +0x95 fp=0x7fff534ae7f8 sp=0x7fff534ae7b0 pc=0x417785
runtime.(*mcentral).cacheSpan(0x2cffd80, 0x7fd1a382d000)
        /usr/local/go/src/runtime/mcentral.go:106 +0x2ff fp=0x7fff534ae858 sp=0x7fff534ae7f8 pc=0x41728f
runtime.(*mcache).refill(0x7fd1a382d008, 0x2a)
        /usr/local/go/src/runtime/mcache.go:135 +0x86 fp=0x7fff534ae878 sp=0x7fff534ae858 pc=0x416d26
runtime.(*mcache).nextFree(0x7fd1a382d008, 0x2cf6e2a, 0x7fd1a382d008, 0x7fd1a382d000, 0x8)
        /usr/local/go/src/runtime/malloc.go:786 +0x88 fp=0x7fff534ae8b0 sp=0x7fff534ae878 pc=0x40b648
runtime.mallocgc(0x180, 0x190fb40, 0x1, 0x2d17920)
        /usr/local/go/src/runtime/malloc.go:939 +0x76e fp=0x7fff534ae950 sp=0x7fff534ae8b0 pc=0x40bf5e
runtime.newobject(0x190fb40, 0x4000)
        /usr/local/go/src/runtime/malloc.go:1068 +0x38 fp=0x7fff534ae980 sp=0x7fff534ae950 pc=0x40c368
runtime.malg(0x74ac0e00008000, 0x2d015f0)
        /usr/local/go/src/runtime/proc.go:3220 +0x31 fp=0x7fff534ae9c0 sp=0x7fff534ae980 pc=0x436871
runtime.mpreinit(...)
        /usr/local/go/src/runtime/os_linux.go:311
runtime.mcommoninit(0x2cf9240)
        /usr/local/go/src/runtime/proc.go:618 +0xc2 fp=0x7fff534ae9f8 sp=0x7fff534ae9c0 pc=0x430222
runtime.schedinit()
        /usr/local/go/src/runtime/proc.go:540 +0x74 fp=0x7fff534aea50 sp=0x7fff534ae9f8 pc=0x42feb4
runtime.rt0_go(0x7fff534aea88, 0x3, 0x7fff534aea88, 0x0, 0x0, 0x3, 0x7fff534aee07, 0x7fff534aee0f, 0x7fff534aee13, 0x0, ...)
        /usr/local/go/src/runtime/asm_amd64.s:195 +0x11a fp=0x7fff534aea58 sp=0x7fff534aea50 pc=0x458c4a
</code></pre>

Promise Preston
<p>I was able to fix the issue, it was due to insufficient memory on the bastion host.</p>
<p><strong>Here's how I fixed it</strong>:</p>
<p>First I ran the command below to check the memory utilization:</p>
<pre><code>free -m
</code></pre>
<p>which gave me the following output which confirmed that the insufficient memory:</p>
<pre><code>              total        used        free      shared  buff/cache   available
Mem:           3.8G        3.7G         93M        988K         34M         27M
Swap:            0B          0B          0B
</code></pre>
<p>You can temporarily run the command below to free up little memory space:</p>
<pre><code>free &amp;&amp; sync &amp;&amp; echo 3 &gt; /proc/sys/vm/drop_caches &amp;&amp; free
</code></pre>
<p>What I did next was to upgrade the EC2 Instance type of the Bastion Host since it's an AWS EC2 Instance from <strong>t2.medium (4GB)</strong> to <strong>t2.large (8GB)</strong> using the steps stated here: <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-resize.html" rel="nofollow noreferrer">Change the instance type</a>. Basically, all you have to do is to:</p>
<ul>
<li>choose <strong>Instance state</strong> &gt; <strong>Stop instance</strong></li>
<li>choose <strong>Actions</strong> &gt; <strong>Instance settings</strong> &gt; <strong>Change instance type</strong></li>
<li>select a new instance type with higher capacity and choose <strong>Apply</strong></li>
<li>choose <strong>Instance state</strong> &gt; <strong>Start instance</strong></li>
</ul>
<p>Wait for a few minutes and everything should take effect. Please take note that your <strong>Public IP address</strong> for the Bastion Host might change after the upgrade.</p>
<p><strong>Reference</strong>:  <a href="https://github.com/kubeflow/kubeflow/issues/3803#issuecomment-517243802" rel="nofollow noreferrer">Kubeflow deployment error &quot;fatal error: runtime: out of memory&quot; #3803</a></p>
