<p>Sorry to ask this, I am relatively new in Kubernetes and Ceph, only have a little idea about this.</p>

<p>I have setup Kubernetes and Ceph using this tutorial(<a href="http://tutorial.kubernetes.noverit.com/content/ceph.html" rel="nofollow noreferrer">http://tutorial.kubernetes.noverit.com/content/ceph.html</a>)</p>

<p>I had set up my cluster like this:</p>

<p>1 Kube-Master and 2 worker Nodes(this acts ceph monitor with 2 OSD in each)<br>
The ceph-deploy I used to setup ceph cluster is in the Kube-master.</p>

<p>Everything is working fine, I installed my sample web application(deployment)  with 5 replicas, which will create a file when the rest API is hit. The file is getting copied to every node.</p>

<p>But after 10 min, I created one more file using the API, but when I try to list(ls -l) I am getting the following error:</p>

<p>For node1:</p>

<pre><code>ls: cannot access 'previousFile.txt': Structure needs cleaning
previousFile.txt  newFile.txt
</code></pre>

<p>For node2:</p>

<pre><code>previousFile.txt  
</code></pre>

<p>For node2 the new file is not created </p>

<p>What might be the issue? I have tried many times still same error pop up. </p>

<p>Any help appreciated.</p>

JibinNajeeb
<p>This totally looks like your filesystem got corrupted. Things to check:</p>

<ul>
<li><code>$ kubectl logs &lt;ceph-pod1&gt;</code></li>
<li><code>$ kubectl logs &lt;ceph-pod2&gt;</code></li>
<li><code>$ kubectl describe deployment &lt;ceph-deployment&gt;</code> # did any of the pods restart?</li>
</ul>

<p>Some info about the error message <a href="https://unix.stackexchange.com/questions/330742/cannot-remove-file-structure-needs-cleaning">here</a>.</p>

<p>Depending on what you have you might need to start from scratch. Or you can take a look a <a href="https://ceph.com/planet/recovering-data-from-a-rbd-image/" rel="nofollow noreferrer">recovering data</a> in Ceph, but may not work if you don't have a snap.</p>

<p>Running Ceph on Kubernetes can be very tricky because any start/restart for a specific node starting on a different Kubernetes node might corrupt the data, so you need to make sure that part is pretty solid possibly using <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/" rel="nofollow noreferrer">Node Affinity</a> or running Ceph pods on specific Kubernetes nodes with <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/" rel="nofollow noreferrer">labels</a>.</p>
