<p>I have deployed cockroachdb with a <a href="https://github.com/helm/charts/tree/master/stable/cockroachdb" rel="nofollow noreferrer">stable helm chart</a>.
Unfortunately, I didn't realize the default conf gives me a very small 1Gi, unresizable persistent volume.
I also didn't realize that the cockroachdb was using quite a lot of space to monitor itself with time series.</p>

<p>Now, my persistent volumes are full, my cockroachdb pods are crashing:</p>

<pre><code>log: exiting because of error: log: cannot create log: open /cockroach/cockroach-data/logs/cockroach.ckdb-cockroachdb-0.root.2018-09-09T14_53_47Z.000001.log: no space left on device
</code></pre>

<p>And I can't resize the volume:</p>

<pre><code>kubectl patch pvc datadir-ckdb-cockroachdb-0 -p '{"spec":{"resources":{"requests":{"storage":"10Gi"}}}}'
The PersistentVolumeClaim "datadir-ckdb-cockroachdb-0" is invalid: spec: Forbidden: field is immutable after creation
</code></pre>

<p>Now I'm stuck as I can't run a node to get my data back.
Is there anyway out of this? I would at least like to retrieve my data. My service is anyway crashed.</p>

<p>Second question: if I want to avoid this in the future, what values should be used to have dynamically resizable volumes on GKE?</p>

<p>Third question: should the default in the helm chart really stay like that?</p>

<p>As <a href="https://stackoverflow.com/users/9231144/patrick-w">https://stackoverflow.com/users/9231144/patrick-w</a> mentioned, automatic resize of the volumes isn't possible until Kubernetes/GKE version 1.11.</p>

<p>In the meantime, it is possible to manually resize them by editing the disks in the <a href="https://console.cloud.google.com/compute/disks" rel="nofollow noreferrer">GCE management console</a>. Go there, click on the disks you want to resize, click the Edit button near the top of the page, type in the new desired size of the disk in GB, and click "Save". You'll then have to SSH into the relevant pods (e.g. <code>kubectl exec -it ckdb-cockroachdb-0 bash</code>) and resize the filesystem to use the new disk capacity with a command like <code>resize2fs</code>.</p>

<p>As for your question about changing the default disk size in the Helm Chart, it's a fair question. But what would a good default size be? Too low, and it's easy for this to happen. Too high, and it won't work in environments that don't have large enough disks for the deployment to succeed. In particular, <code>minikube</code> uses tmpfs-backed volumes, so their size is quite limited by the memory of your machine. At the very least, warning in the output after instantiating the Chart seems warranted.</p>
