<p>I would like the containers in my pod to share a volume for temporary (cached) data. I don't mind if the data is lost when the pod terminates (in fact, I want the data deleted and space reclaimed).</p>

<p>The <a href="https://kubernetes.io/docs/concepts/storage/volumes/#emptydir" rel="noreferrer">kubernetes docs</a> make an <code>emptyDir</code> sound like what I want:</p>

<blockquote>
  <p>An emptyDir volume is first created when a Pod is assigned to a Node, and exists as long as that Pod is running on that node</p>
</blockquote>

<p>.. and </p>

<blockquote>
  <p>By default, emptyDir volumes are stored on whatever medium is backing the node - that might be disk or SSD or network storage, depending on your environment. However, you can set the emptyDir.medium field to "Memory" to tell Kubernetes to mount a tmpfs (RAM-backed filesystem) for you instead</p>
</blockquote>

<p>That sounds like the default behaviour is to store the volume on disk, unless I explicitly request in-memory.</p>

<p>However, if I create the following pod on my GKE cluster:</p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: alpine
spec:
  containers:
  - name: alpine
    image: alpine:3.7
    command: ["/bin/sh", "-c", "sleep 60m"]
    volumeMounts:
      - name: foo
        mountPath: /foo
  volumes:
  - name: foo
    emptyDir: {}
</code></pre>

<p>.. and then open a shell on the pod and write a 2Gb file to the volume:</p>

<pre><code>kubectl exec -it alpine -- /bin/sh
$ cd foo/
$ dd if=/dev/zero of=file.txt count=2048 bs=1048576
</code></pre>

<p>Then I can see in the GKE web console that the RAM usage of the container has increased by 2Gb:</p>

<p><a href="https://i.stack.imgur.com/bryZy.png" rel="noreferrer"><img src="https://i.stack.imgur.com/bryZy.png" alt="memory increase in the alpine contianer"></a></p>

<p>It looks to me like the GKE stores <code>emptyDir</code> volumes in memory by default. The workload I plan to run needs plenty of memory, so I'd like the <code>emptyDir</code> volume to be backed by disk - is that possible? The <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/volumes" rel="noreferrer">GKE storage docs</a> don't have much to say on the issue.</p>

<p>An alternative approach might be to use a <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/local-ssd" rel="noreferrer">local SSD</a> for my cached data, however if I mount them as recommended in the GKE docs they're shared by all pods running on the same node and the data isn't cleaned up on pod termination, which doesn't meet my goals of automatically managed resources.</p>

<h2>Mounts</h2>

<p>Here's the output of <code>df -h</code> inside the container:</p>

<pre><code># df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  96.9G     26.2G     70.7G  27% /
overlay                  96.9G     26.2G     70.7G  27% /
tmpfs                     7.3G         0      7.3G   0% /dev
tmpfs                     7.3G         0      7.3G   0% /sys/fs/cgroup
/dev/sda1                96.9G     26.2G     70.7G  27% /foo
/dev/sda1                96.9G     26.2G     70.7G  27% /dev/termination-log
/dev/sda1                96.9G     26.2G     70.7G  27% /etc/resolv.conf
/dev/sda1                96.9G     26.2G     70.7G  27% /etc/hostname
/dev/sda1                96.9G     26.2G     70.7G  27% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                     7.3G     12.0K      7.3G   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                     7.3G         0      7.3G   0% /proc/kcore
tmpfs                     7.3G         0      7.3G   0% /proc/timer_list
tmpfs                     7.3G         0      7.3G   0% /proc/sched_debug
tmpfs                     7.3G         0      7.3G   0% /sys/firmware
</code></pre>

<h2>The View from the Node</h2>

<p>I discovered it's possible to ssh into the node instance, and I was able to find the 2Gb file on the node filesystem:</p>

<pre><code>root@gke-cluster-foo-pool-b-22bb9925-xs5p:/# find . -name file.txt
./var/lib/kubelet/pods/79ad1aa4-4441-11e8-af32-42010a980039/volumes/kubernetes.io~empty-dir/foo/file.txt
</code></pre>

<p>Now that I can see it is being written to the underlying filesystem, I'm wondering if maybe the RAM usage I'm seeing in the GKE web UI is the linux filesystem cache or similar, rather than the file being stored in a RAM disk?</p>

<p>From the mount information you've supplied, the <code>emptyDir</code> volume is mounted on a drive partition, so it's working as intended, and isn't mounted in memory. It's likely that the memory usage you see is due to the filesystem buffer cache, so with sufficient memory pressure, it'd eventually get written to the disk. However, given that you have so much free memory, it's likely that the system saw no need to do so immediately.</p>

<p>If you have more doubts, give <code>sync</code> or <code>echo 3 &gt; /proc/sys/vm/drop_caches</code> a go on the machines to flush filesystem information to disk. You should see a change in memory usage.</p>
