<p>We want to deploy a k8s cluster which will run ~100 IO-heavy pods at the same time. They should all be able to access the same volume.</p>

<p>What we tried so far:</p>

<ul>
<li>CephFS

<ul>
<li>was very complicated to set up. Hard to troubleshoot. In the end, it crashed a lot and the cause was not entirely clear.</li>
</ul></li>
<li>Helm NFS Server Provisioner

<ul>
<li>runs pretty well, but when IO peaks a single replica is not enough. We could not get multiple replicas to work at all.</li>
</ul></li>
<li>MinIO

<ul>
<li>is a great tool to create storage buckets in k8s. But our operations require fs mounting. That is theoretically possible with s3fs, but since we run ~100 pods, we would need to run 100 s3fs sidecars additionally. Thats seems like a bad idea.</li>
</ul></li>
</ul>

<p>There has to be some way to get 2TB of data mounted in a GKE cluster with relatively high availability?  </p>

<p>Firestorage seems to work, but it's a magnitude more expensive than other solutions, and with a lot of IO operations it quickly becomes infeasible.</p>

<hr>

<p><em>I contemplated creating this question on server fault, but the k8s community is a lot smaller than SO's.</em></p>

<p>I think I have a definitive answer as of Jan 2020, at least for our usecase:</p>

<pre><code>| Solution        | Complexity | Performance | Cost           |
|-----------------|------------|-------------|----------------|
| NFS             | Low        | Low         | Low            |
| Cloud Filestore | Low        | Mediocre?   | Per Read/Write |
| CephFS          | High*      | High        | Low            |

* You need to add an additional step for GKE: Change the base image to ubuntu
</code></pre>

<p>I haven't benchmarked Filestore myself, but I'll just go with <a href="https://stackoverflow.com/users/1420941/stringy05">stringy05</a>'s response: <em>others have trouble getting really good throughput from it</em></p>

<p>Ceph could be a lot easier if it was supported by Helm.</p>
