title: HDFS complains "Resources are low on NN" and enters safemode while I have enough free space
body: <p>I am using hadoop-2.6.0 on Ubuntu 16.04.3 LTS, and everything went well in my test env. But while running in the production env, my HDFS repeatedly complains about not having enough space for namenode, and HDFS has entered safemode.</p>
  
  <blockquote>
    <p>2017-11-25 14:00:48,731 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on available disk space. Already in safe mode.</p>
    
    <p>2017-11-25 14:00:48,731 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode is ONResources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE:
     If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.</p>
    
    <p>2017-11-25 14:00:53,732 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space available on volume '/home/hadoop/.Private' is 0, which is below the configured reserved a
    mount 104857600</p>
  </blockquote>
  
  <p>But the truth is I have more than enough space</p>
  
  <blockquote>
    <p>Filesystem                       Size  Used Avail Use% Mounted on</p>
    
    <p>udev                              63G     0   63G   0% /dev</p>
    
    <p>tmpfs                             13G   33M   13G   1% /run</p>
    
    <p>/dev/mapper/hdfsmaster--vg-root  7.1T  1.3T  5.6T  19% /</p>
    
    <p>tmpfs                             63G  4.0K   63G   1% /dev/shm</p>
    
    <p>tmpfs                            5.0M     0  5.0M   0% /run/lock</p>
    
    <p>tmpfs                             63G     0   63G   0% /sys/fs/cgroup</p>
    
    <p>/dev/sda2                        473M   55M  394M  13% /boot</p>
    
    <p>/dev/sda1                        511M  4.0K  511M   1% /boot/efi</p>
    
    <p>/home/hadoop/.Private            7.1T  1.3T  5.6T  19% /home/hadoop</p>
  </blockquote>
  
  <p>HDFS data/name directories are all in /home/hadoop directory</p>
  
  <p>HDFS can be started successfully, and it's NOT in safemode at beginning. But after some time (like 5, 6 hours), even if I did not perform any operation (read / write / query ), it will complain about lacking of space and enter safemode automatically.</p>
  
  <p>I have tried:</p>
  
  <ol>
  <li>restart hdfs</li>
  <li>restart the server</li>
  <li>re-format namenode</li>
  <li>change data/name folder and restart hdfs</li>
  </ol>
  
  <p>none of the above method helps.</p>
  
  <p>Please help! This is killing me!</p>