<p>So I am running Django App in kubernetes pod at when trying to save an image file:</p>
<pre><code>img_obj.image.save(img_file, File(img_file_org))
</code></pre>
<p>I am getting no space left error:</p>
<pre><code>  File &quot;/code/ocr_client/management/commands/pdf_to_image.py&quot;, line 126, in handle
    img_obj.image.save(img_file, File(img_file_org))
  File &quot;/opt/conda/lib/python3.7/site-packages/django/db/models/fields/files.py&quot;, line 88, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File &quot;/opt/conda/lib/python3.7/site-packages/django/core/files/storage.py&quot;, line 54, in save
    return self._save(name, content)
  File &quot;/opt/conda/lib/python3.7/site-packages/django/core/files/storage.py&quot;, line 274, in _save
    fd = os.open(full_path, self.OS_OPEN_FLAGS, 0o666)
OSError: [Errno 28] No space left on device: '/code/pca_back_data/media/file1.png'
</code></pre>
<p>I already ran</p>
<pre><code>kubectl exec &lt;my-pod&gt; -- df -ah
</code></pre>
<p>And there is still 20% of the space left (100GB)</p>
<p>I also ran as suggested in other thread:</p>
<pre><code>kubectl exec &lt;my-pod&gt; -- df -hi
</code></pre>
<p>and the usage of inodes was only 5%</p>
<p>I am not sure what else might be the issue here?
Is there some config in Kubernetes that restricts usage of storage for a pod/process?</p>

Alex T
<p>If you are getting the &quot;<code>No space left on device</code>&quot; error even when the disk usage and inode usage are low, it might be that the disk resources for your specific pod are limited. The Kubernetes system can set limits on resources like CPU, memory, and disk storage.</p>
<p>So start with checking the Kubernetes resource limits and requests for your pod: run <code>kubectl describe pod &lt;my-pod&gt;</code> to check if there are resource limits or requests set. Look for something like:</p>
<pre class="lang-yaml prettyprint-override"><code>Resources:
  Limits:
    ephemeral-storage: 1Gi
  Requests:
    ephemeral-storage: 500Mi
</code></pre>
<p>The <a href="https://docs.openshift.com/container-platform/4.13/storage/understanding-ephemeral-storage.html" rel="nofollow noreferrer"><code>ephemeral-storage</code></a> represents the storage available for your pod to use. If it is set too low, you might need to adjust it.</p>
<p>Try also to set said resource requests and limits yourself: You can specify the resources available for your pod by adding the following in your pod or deployment configuration with:</p>
<pre class="lang-yaml prettyprint-override"><code>resources:
  requests:
    ephemeral-storage: &quot;1Gi&quot;
  limits:
    ephemeral-storage: &quot;2Gi&quot;
</code></pre>
<p>That allows your pod to request 1 GiB of ephemeral storage and limit it to using 2 GiB. Adjust these values as needed based on the size of the images you are dealing with.</p>
<hr />
<p>But another approach would be to consider using Persistent Volumes (PV): If your application needs to store a lot of data (like many large image files), consider using a <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/" rel="nofollow noreferrer">Persistent Volume (PV) and Persistent Volume Claim (PVC)</a>. PVs represent physical storage in a cluster and can be used to provision durable storage resources. You would need to change your application's code or configuration to write to this PV.</p>
<p>Define a PV and PVC:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: &quot;/mnt/data&quot;

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
</code></pre>
<p>And in your pod spec, you would add:</p>
<pre class="lang-yaml prettyprint-override"><code>volumes:
  - name: my-storage
    persistentVolumeClaim:
      claimName: my-pvc
</code></pre>
<p>And mount it into your pod:</p>
<pre class="lang-yaml prettyprint-override"><code>volumeMounts:
  - mountPath: &quot;/code/pca_back_data/media&quot;
    name: my-storage
</code></pre>
<p>When you create the Persistent Volume (PV) and mount it into your pod at the same location (<code>/code/pca_back_data/media</code>), your application will continue to write to the same directory without needing to change the Django settings.</p>
<p>The only difference is that the storage will now be backed by a Persistent Volume which is designed to handle larger amounts of data and will not be subject to the same restrictions as the pod's ephemeral storage.</p>
<p>In that case, no changes would be required in your Django settings. The application would continue to write to the same path but the underlying storage mechanism would have changed.</p>
<p>However, do note that <code>hostPath</code> should be used only for development or testing. For production, consider using a networked storage like an NFS server, or a cloud provider's storage service.</p>
<hr />
<blockquote>
<p>I am already using PVC that is attached to this pod. And it has more than enough storage. What is stranger is that not all files are failing this… –</p>
</blockquote>
<p>As I commented, it could be a concurrency issue: If multiple processes or threads are trying to write to the same file or location simultaneously, that might cause some operations to fail with &quot;<code>No space left on device</code>&quot; errors.<br />
Also, Although the PVC has enough available space, individual filesystems on the PVC might have quotas that limit how much space they can use. Check if there are any such quotas set on your filesystem.</p>
<p>The OP confirms:</p>
<blockquote>
<p>There is something like this happening - multiple processes are using the same PVC directory, maybe not exactly same file but same parent directory can be accessed by those processes.</p>
</blockquote>
<p>Multiple processes using the same PVC directory or parent directory should generally not be a problem, as long as they are not trying to write to the same file at the same time. But if these processes are creating a large number of files or very large files, and if your PVC or underlying filesystem has a limit on the number of files (inodes) or the total size of files it can handle, that could potentially lead to the &quot;No space left on device&quot; error.</p>
<p>You can check for filesystem quotas on a PVC:</p>
<ul>
<li><p>Connect to your pod: <code>kubectl exec -it &lt;your-pod&gt; -- /bin/bash</code></p>
</li>
<li><p>Install the <code>quota</code> package: This can usually be done with <a href="https://doc.ubuntu-fr.org/quota" rel="nofollow noreferrer"><code>apt-get install quota</code></a> on Debian/Ubuntu systems or <code>yum install quota</code> on CentOS/RHEL systems. If these commands do not work, you may need to look up how to install <code>quota</code> for your specific container's operating system.</p>
</li>
<li><p>Check quotas: Run <code>quota -v</code> to view quota information. If quotas are enabled and you are nearing or at the limit, you will see that here.</p>
</li>
</ul>
<p>If your filesystem does not support quotas or they are not enabled, you will not get useful output from <code>quota -v</code>. In that case, or if you are unable to install the <code>quota</code> package, you might need to check for quotas from outside the pod, which would depend on your Kubernetes setup and cloud provider.</p>
<p>If you are still having trouble, another possible culprit could be a Linux kernel parameter called <a href="https://unix.stackexchange.com/q/444998/7490"><code>fs.inotify.max_user_watches</code></a>, which can limit the number of files the system can monitor for changes. If you are opening and not properly closing a large number of files, you could be hitting this limit. You can check its value with <code>cat /proc/sys/fs/inotify/max_user_watches</code> and increase it if necessary.</p>
<hr />
<p>The OP adds:</p>
<blockquote>
<p>I think the issue in my case is that <code>/tmp</code> folder inside the pod is running out of space (in Django <code>/tmp</code> is used for the files when saving to database if I understand correctly), not sure how to expand size of it?</p>
</blockquote>
<p>Yes, you're correct. Django, like many other systems, uses the <code>/tmp</code> directory to handle temporary files, which includes processing file uploads. If the <code>/tmp</code> directory is running out of space, you can consider the following options:</p>
<ul>
<li>Increase the Pod ephemeral storage limit: as mentioned above, you can adjust the ephemeral storage requests and limits in your pod or deployment configuration, like so:</li>
</ul>
<pre class="lang-yaml prettyprint-override"><code>resources:
  requests:
    ephemeral-storage: &quot;2Gi&quot;  # Request 2Gi of ephemeral storage
  limits:
    ephemeral-storage: &quot;4Gi&quot;  # Limit ephemeral storage usage to 4Gi
</code></pre>
<p>Remember to adjust these values according to your needs.</p>
<ul>
<li>Or use an <code>emptyDir</code> Volume for <code>/tmp</code>: meaning use a Kubernetes <code>emptyDir</code> volume for your <code>/tmp</code> directory. When a Pod is assigned to a Node, Kubernetes will create an <code>emptyDir</code> volume for that Pod, and it will exist as long as that Pod is running on that node. The <code>emptyDir</code> volume can use the node's storage space, and you can specify a size limit.</li>
</ul>
<p>Here is how you might define an <code>emptyDir</code> volume for <code>/tmp</code>:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: my-image
    volumeMounts:
    - name: tmp-storage
      mountPath: /tmp
  volumes:
  - name: tmp-storage
    emptyDir:
      medium: &quot;Memory&quot;
      sizeLimit: &quot;2Gi&quot;  # Set a size limit for the volume
</code></pre>
<p>The <code>medium: &quot;Memory&quot;</code> means that the <code>emptyDir</code> volume is backed by memory (tmpfs) instead of disk storage. If you remove this line, the <code>emptyDir</code> volume will use disk storage. The <code>sizeLimit</code> is optional.</p>
<ul>
<li>You can also consider using a dedicated PVC for <code>/tmp</code>: If the above options are not feasible or if you need more control over the storage for <code>/tmp</code>, you can also use a dedicated PVC for it, similar to the one you're using for <code>/code/pca_back_data/media</code>.</li>
</ul>
<p>Remember that changes to your pod or deployment configuration need to be applied with <code>kubectl apply -f &lt;configuration-file&gt;</code>, and you may need to recreate your pod or deployment for the changes to take effect.</p>
<hr />
<p>The OP concludes in <a href="https://stackoverflow.com/questions/76570871/django-on-kubernetes-pod-no-space-left-on-device/76636516?noredirect=1#comment135345270_76636516">the comments</a>:</p>
<blockquote>
<p>I managed to solve this issue: looks like the GCP storage disc was somehow corrupted and we changed to another and it seems to be fine now.</p>
</blockquote>
