<p>A few months back, I deployed the Elastic-Search (version - 8.0.1) on Kubernetes (GCP) as a service as External load balancer using this <a href="https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-deploy-elasticsearch.html" rel="nofollow noreferrer">guide</a>.</p>
<p>Now, I am unable to perform any read or write operation on ElasticSearch. I checked the logs, in which I found that memory of the node is almost full.</p>
<p><strong>Here are some logs which support this analysis:</strong></p>
<blockquote>
<p>flood stage disk watermark [95%] exceeded on [hulk-es-default-0][/usr/share/elasticsearch/data] free: 18.5mb[1.8%], all indices on this node will be marked read-only</p>
<p>Cluster health status changed from [YELLOW] to [RED] (reason: [shards failed [[1][0]]]).</p>
<p>This node is unhealthy: health check failed on [/usr/share/elasticsearch/data].`</p>
</blockquote>
<p><strong>Here are the errors that are coming when performing any read/write operation:</strong></p>
<blockquote>
<p>elasticsearch.exceptions.TransportError: TransportError(503, 'master_not_discovered_exception', None)</p>
<p>elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPSConnectionPool(host='...', port=****): Read timed out. (read timeout=30))</p>
</blockquote>
<p>I increased the capacity of my elasticsearch persistent volume claim(PVC) but was unable to create the pod with that new volume.</p>
<p>I followed the following steps -</p>
<ul>
<li><p>Set the allowVolumeExpansion field to true in their StorageClass object(s)</p>
</li>
<li><p>Scaled ElasticSearch Operator Deployment to 0 Replicas.</p>
</li>
<li><p>Deleted the statefulset Object without deleting the pods using</p>
<p><code>kubectl delete sts &lt;statefulset-name&gt; --cascade=orphan</code></p>
</li>
</ul>
<p>Before deleting I saved the yaml of the statefulset using</p>
<pre><code>kubectl get sts &lt;statefulset-name&gt; -o yaml
</code></pre>
<ul>
<li>Increased the storage in capacity in the yaml file of PVC.</li>
<li>Recreated the StatefulSet with the new storage request by the yaml file I saved using</li>
</ul>
<p><code>kubectl apply -f file-name.yml</code></p>
<ul>
<li>Scaled back the operator deployment to 1</li>
</ul>
<p>But, when I recreated the stateful set,  the <code>CrashLoopBackOff</code> error is being shown every-time.</p>
<p>Following are some logs -</p>
<ul>
<li>readiness probe failed</li>
<li>Likely root cause: java.io.IOException: No space left on device</li>
<li>using data paths, mounts [[/usr/share/elasticsearch/data (/dev/sdb)]], net usable_space [0b], net total_space [975.8mb], types [ext4]</li>
</ul>
<p>The persistent disk's volume that the ES pod is accessing is increased but still the pod is unable to start.
Can anyone guide me here, what is the problem here ?</p>

<p>It seems, for some reason the Pod is not seeing the new volume Size. Have you tried <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/volume-expansion" rel="nofollow noreferrer">this</a>, for the GKE Volume expansion?</p>
<p>If the Pod is always in the <code>CrashLoopBackOff</code> state you can use e.g: <code>kubectl debug mypod -it --image=busybox</code> So you will attach a debug container to your Pod and check what is going on with the mounted volume.</p>
<p>Others things you can also do is to create a snapshot/backup of your Volume and restore it on a new bigger volume to see if the Issue still persist.</p>
