<p>I'm running Elasicsearch (<code>floragunncom/sg-elasticsearch:7.0.1-oss-35.0.0</code>) on kubernetes cluster(v1.18 - kind).</p>
<p>I've assigned each Elasticsearch data node (ie. pod) with <code>200Mi</code> of PVC as storage. But when the elasticseach cluster(3 master, 3 data, 2 ingest) is up, it is showing the following warning message:</p>
<blockquote>
<p>{&quot;type&quot;: &quot;server&quot;, &quot;timestamp&quot;: &quot;2020-07-20T06:27:14,354+0000&quot;, &quot;level&quot;: &quot;WARN&quot;, &quot;component&quot;: &quot;o.e.c.r.a.DiskThresholdMonitor&quot;, &quot;cluster.name&quot;: &quot;topology-es&quot;, &quot;node.name&quot;: &quot;master-topology-es-1&quot;, &quot;cluster.uuid&quot;: &quot;U0Yf6tiJSHC57R_QHa2qiw&quot;, &quot;node.id&quot;: &quot;tDet2llLQaydoxfYT4TcpQ&quot;,  &quot;message&quot;: &quot;high disk watermark [90%] exceeded on [C-TUE5_cR1ihqrhThpWnsg][data-topology-es-1][/usr/share/elasticsearch/data/nodes/0] free: 5.5gb[7%], shards will be relocated away from this node&quot;  }</p>
</blockquote>
<ul>
<li>Where is this <code>5.5gb[7%]</code> coming from?</li>
</ul>
<p>I aslo checked the shard's status:</p>
<pre class="lang-sh prettyprint-override"><code>$ curl -XGET &quot;https://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason&quot; -u &quot;admin:XXXXX&quot; --insecure
searchguard 0 p STARTED    
searchguard 0 r UNASSIGNED REPLICA_ADDED
searchguard 0 r UNASSIGNED REPLICA_ADDED
</code></pre>
<ul>
<li>If <code>200Mi</code> is not enough, then how comes the primary shard is successfully assigned in <code>data-node-0</code> but the replicas are not in <code>data-node-1</code> and <code>data-node-2</code> respectively?</li>
</ul>
<p>Update:</p>
<pre class="lang-sh prettyprint-override"><code>$ curl -XGET &quot;https://localhost:9200/_nodes/stats/fs?pretty&quot; -u &quot;admin:XXXX&quot; --insecure
{
  &quot;_nodes&quot; : {
    &quot;total&quot; : 8,
    &quot;successful&quot; : 8,
    &quot;failed&quot; : 0
  },
  &quot;cluster_name&quot; : &quot;topology-es&quot;,
  &quot;nodes&quot; : {
    &quot;jYkduRCmQ2GEHMn2DeBrxg&quot; : {
      &quot;timestamp&quot; : 1595231421976,
      &quot;name&quot; : &quot;data-topology-es-0&quot;,
      &quot;transport_address&quot; : &quot;10.244.0.59:9300&quot;,
      &quot;host&quot; : &quot;10.244.0.59&quot;,
      &quot;ip&quot; : &quot;10.244.0.59:9300&quot;,
      &quot;roles&quot; : [
        &quot;data&quot;
      ],
      &quot;fs&quot; : {
        &quot;timestamp&quot; : 1595231421989,
        &quot;total&quot; : {
          &quot;total_in_bytes&quot; : 84054204416,
          &quot;free_in_bytes&quot; : 10231291904,
          &quot;available_in_bytes&quot; : 5917450240
        },
        &quot;data&quot; : [
          {
            &quot;path&quot; : &quot;/usr/share/elasticsearch/data/nodes/0&quot;,
            &quot;mount&quot; : &quot;/usr/share/elasticsearch/data (/dev/sda3)&quot;,
            &quot;type&quot; : &quot;ext4&quot;,
            &quot;total_in_bytes&quot; : 84054204416,
            &quot;free_in_bytes&quot; : 10231291904,
            &quot;available_in_bytes&quot; : 5917450240
          }
        ],
        &quot;io_stats&quot; : {
          &quot;devices&quot; : [
            {
              &quot;device_name&quot; : &quot;sda3&quot;,
              &quot;operations&quot; : 77306,
              &quot;read_operations&quot; : 7745,
              &quot;write_operations&quot; : 69561,
              &quot;read_kilobytes&quot; : 230532,
              &quot;write_kilobytes&quot; : 500284
            }
          ],
          &quot;total&quot; : {
            &quot;operations&quot; : 77306,
            &quot;read_operations&quot; : 7745,
            &quot;write_operations&quot; : 69561,
            &quot;read_kilobytes&quot; : 230532,
            &quot;write_kilobytes&quot; : 500284
          }
        }
      }
    },
    &quot;N7ql_AneQoa6w8yvgBhYZg&quot; : {
      &quot;timestamp&quot; : 1595231421977,
      &quot;name&quot; : &quot;client-topology-es-0&quot;,
      &quot;transport_address&quot; : &quot;10.244.0.49:9300&quot;,
      &quot;host&quot; : &quot;10.244.0.49&quot;,
      &quot;ip&quot; : &quot;10.244.0.49:9300&quot;,
      &quot;roles&quot; : [
        &quot;ingest&quot;
      ],
      &quot;fs&quot; : {
        &quot;timestamp&quot; : 1595231421981,
        &quot;total&quot; : {
          &quot;total_in_bytes&quot; : 84054204416,
          &quot;free_in_bytes&quot; : 10231291904,
          &quot;available_in_bytes&quot; : 5917450240
        },
        &quot;data&quot; : [
          {
            &quot;path&quot; : &quot;/usr/share/elasticsearch/data/nodes/0&quot;,
            &quot;mount&quot; : &quot;/usr/share/elasticsearch/data (/dev/sda3)&quot;,
            &quot;type&quot; : &quot;ext4&quot;,
            &quot;total_in_bytes&quot; : 84054204416,
            &quot;free_in_bytes&quot; : 10231291904,
            &quot;available_in_bytes&quot; : 5917450240
          }
        ],
        &quot;io_stats&quot; : {
          &quot;devices&quot; : [
            {
              &quot;device_name&quot; : &quot;sda3&quot;,
              &quot;operations&quot; : 84052,
              &quot;read_operations&quot; : 7853,
              &quot;write_operations&quot; : 76199,
              &quot;read_kilobytes&quot; : 231804,
              &quot;write_kilobytes&quot; : 565340
            }
          ],
          &quot;total&quot; : {
            &quot;operations&quot; : 84052,
            &quot;read_operations&quot; : 7853,
            &quot;write_operations&quot; : 76199,
            &quot;read_kilobytes&quot; : 231804,
            &quot;write_kilobytes&quot; : 565340
          }
        }
      }
    },
    &quot;gse01W_XR7qKvZPHPq7n7g&quot; : {
      &quot;timestamp&quot; : 1595231421984,
      &quot;name&quot; : &quot;master-topology-es-2&quot;,
      &quot;transport_address&quot; : &quot;10.244.0.57:9300&quot;,
      &quot;host&quot; : &quot;10.244.0.57&quot;,
      &quot;ip&quot; : &quot;10.244.0.57:9300&quot;,
      &quot;roles&quot; : [
        &quot;master&quot;
      ],
      &quot;fs&quot; : {
        &quot;timestamp&quot; : 1595231421993,
        &quot;total&quot; : {
          &quot;total_in_bytes&quot; : 84054204416,
          &quot;free_in_bytes&quot; : 10231291904,
          &quot;available_in_bytes&quot; : 5917450240
        },
        &quot;data&quot; : [
          {
            &quot;path&quot; : &quot;/usr/share/elasticsearch/data/nodes/0&quot;,
            &quot;mount&quot; : &quot;/usr/share/elasticsearch/data (/dev/sda3)&quot;,
            &quot;type&quot; : &quot;ext4&quot;,
            &quot;total_in_bytes&quot; : 84054204416,
            &quot;free_in_bytes&quot; : 10231291904,
            &quot;available_in_bytes&quot; : 5917450240
          }
        ],
        &quot;io_stats&quot; : {
          &quot;devices&quot; : [
            {
              &quot;device_name&quot; : &quot;sda3&quot;,
              &quot;operations&quot; : 77627,
              &quot;read_operations&quot; : 7747,
              &quot;write_operations&quot; : 69880,
              &quot;read_kilobytes&quot; : 230560,
              &quot;write_kilobytes&quot; : 504608
            }
          ],
          &quot;total&quot; : {
            &quot;operations&quot; : 77627,
            &quot;read_operations&quot; : 7747,
            &quot;write_operations&quot; : 69880,
            &quot;read_kilobytes&quot; : 230560,
            &quot;write_kilobytes&quot; : 504608
          }
        }
      }
    },
    &quot;bBNEuz2pSkWeOUmDIowEDQ&quot; : {
      &quot;timestamp&quot; : 1595231421991,
      &quot;name&quot; : &quot;client-topology-es-1&quot;,
      &quot;transport_address&quot; : &quot;10.244.0.51:9300&quot;,
      &quot;host&quot; : &quot;10.244.0.51&quot;,
      &quot;ip&quot; : &quot;10.244.0.51:9300&quot;,
      &quot;roles&quot; : [
        &quot;ingest&quot;
      ],
      &quot;fs&quot; : {
        &quot;timestamp&quot; : 1595231421995,
        &quot;total&quot; : {
          &quot;total_in_bytes&quot; : 84054204416,
          &quot;free_in_bytes&quot; : 10231291904,
          &quot;available_in_bytes&quot; : 5917450240
        },
        &quot;data&quot; : [
          {
            &quot;path&quot; : &quot;/usr/share/elasticsearch/data/nodes/0&quot;,
            &quot;mount&quot; : &quot;/usr/share/elasticsearch/data (/dev/sda3)&quot;,
            &quot;type&quot; : &quot;ext4&quot;,
            &quot;total_in_bytes&quot; : 84054204416,
            &quot;free_in_bytes&quot; : 10231291904,
            &quot;available_in_bytes&quot; : 5917450240
          }
        ],
        &quot;io_stats&quot; : {
          &quot;devices&quot; : [
            {
              &quot;device_name&quot; : &quot;sda3&quot;,
              &quot;operations&quot; : 82173,
              &quot;read_operations&quot; : 7788,
              &quot;write_operations&quot; : 74385,
              &quot;read_kilobytes&quot; : 231144,
              &quot;write_kilobytes&quot; : 546544
            }
          ],
          &quot;total&quot; : {
            &quot;operations&quot; : 82173,
            &quot;read_operations&quot; : 7788,
            &quot;write_operations&quot; : 74385,
            &quot;read_kilobytes&quot; : 231144,
            &quot;write_kilobytes&quot; : 546544
          }
        }
      }
    },
    &quot;-hwsv3rqQeSVdfOsKqzrwA&quot; : {
      &quot;timestamp&quot; : 1595231422013,
      &quot;name&quot; : &quot;master-topology-es-0&quot;,
      &quot;transport_address&quot; : &quot;10.244.0.53:9300&quot;,
      &quot;host&quot; : &quot;10.244.0.53&quot;,
      &quot;ip&quot; : &quot;10.244.0.53:9300&quot;,
      &quot;roles&quot; : [
        &quot;master&quot;
      ],
      &quot;fs&quot; : {
        &quot;timestamp&quot; : 1595231422080,
        &quot;total&quot; : {
          &quot;total_in_bytes&quot; : 84054204416,
          &quot;free_in_bytes&quot; : 10231291904,
          &quot;available_in_bytes&quot; : 5917450240
        },
        &quot;data&quot; : [
          {
            &quot;path&quot; : &quot;/usr/share/elasticsearch/data/nodes/0&quot;,
            &quot;mount&quot; : &quot;/usr/share/elasticsearch/data (/dev/sda3)&quot;,
            &quot;type&quot; : &quot;ext4&quot;,
            &quot;total_in_bytes&quot; : 84054204416,
            &quot;free_in_bytes&quot; : 10231291904,
            &quot;available_in_bytes&quot; : 5917450240
          }
        ],
        &quot;io_stats&quot; : {
          &quot;devices&quot; : [
            {
              &quot;device_name&quot; : &quot;sda3&quot;,
              &quot;operations&quot; : 79542,
              &quot;read_operations&quot; : 7762,
              &quot;write_operations&quot; : 71780,
              &quot;read_kilobytes&quot; : 230792,
              &quot;write_kilobytes&quot; : 521656
            }
          ],
          &quot;total&quot; : {
            &quot;operations&quot; : 79542,
            &quot;read_operations&quot; : 7762,
            &quot;write_operations&quot; : 71780,
            &quot;read_kilobytes&quot; : 230792,
            &quot;write_kilobytes&quot; : 521656
          }
        }
      }
    },
    &quot;C-TUE5_cR1ihqrhThpWnsg&quot; : {
      &quot;timestamp&quot; : 1595231421984,
      &quot;name&quot; : &quot;data-topology-es-1&quot;,
      &quot;transport_address&quot; : &quot;10.244.0.61:9300&quot;,
      &quot;host&quot; : &quot;10.244.0.61&quot;,
      &quot;ip&quot; : &quot;10.244.0.61:9300&quot;,
      &quot;roles&quot; : [
        &quot;data&quot;
      ],
      &quot;fs&quot; : {
        &quot;timestamp&quot; : 1595231422003,
        &quot;total&quot; : {
          &quot;total_in_bytes&quot; : 84054204416,
          &quot;free_in_bytes&quot; : 10231291904,
          &quot;available_in_bytes&quot; : 5917450240
        },
        &quot;data&quot; : [
          {
            &quot;path&quot; : &quot;/usr/share/elasticsearch/data/nodes/0&quot;,
            &quot;mount&quot; : &quot;/usr/share/elasticsearch/data (/dev/sda3)&quot;,
            &quot;type&quot; : &quot;ext4&quot;,
            &quot;total_in_bytes&quot; : 84054204416,
            &quot;free_in_bytes&quot; : 10231291904,
            &quot;available_in_bytes&quot; : 5917450240
          }
        ],
        &quot;io_stats&quot; : {
          &quot;devices&quot; : [
            {
              &quot;device_name&quot; : &quot;sda3&quot;,
              &quot;operations&quot; : 77169,
              &quot;read_operations&quot; : 7736,
              &quot;write_operations&quot; : 69433,
              &quot;read_kilobytes&quot; : 230396,
              &quot;write_kilobytes&quot; : 498544
            }
          ],
          &quot;total&quot; : {
            &quot;operations&quot; : 77169,
            &quot;read_operations&quot; : 7736,
            &quot;write_operations&quot; : 69433,
            &quot;read_kilobytes&quot; : 230396,
            &quot;write_kilobytes&quot; : 498544
          }
        }
      }
    },
    &quot;zlL7F5kRTLWWogzffZXomA&quot; : {
      &quot;timestamp&quot; : 1595231422000,
      &quot;name&quot; : &quot;data-topology-es-2&quot;,
      &quot;transport_address&quot; : &quot;10.244.0.63:9300&quot;,
      &quot;host&quot; : &quot;10.244.0.63&quot;,
      &quot;ip&quot; : &quot;10.244.0.63:9300&quot;,
      &quot;roles&quot; : [
        &quot;data&quot;
      ],
      &quot;fs&quot; : {
        &quot;timestamp&quot; : 1595231422001,
        &quot;total&quot; : {
          &quot;total_in_bytes&quot; : 84054204416,
          &quot;free_in_bytes&quot; : 10231291904,
          &quot;available_in_bytes&quot; : 5917450240
        },
        &quot;data&quot; : [
          {
            &quot;path&quot; : &quot;/usr/share/elasticsearch/data/nodes/0&quot;,
            &quot;mount&quot; : &quot;/usr/share/elasticsearch/data (/dev/sda3)&quot;,
            &quot;type&quot; : &quot;ext4&quot;,
            &quot;total_in_bytes&quot; : 84054204416,
            &quot;free_in_bytes&quot; : 10231291904,
            &quot;available_in_bytes&quot; : 5917450240
          }
        ],
        &quot;io_stats&quot; : {
          &quot;devices&quot; : [
            {
              &quot;device_name&quot; : &quot;sda3&quot;,
              &quot;operations&quot; : 77048,
              &quot;read_operations&quot; : 7736,
              &quot;write_operations&quot; : 69312,
              &quot;read_kilobytes&quot; : 230396,
              &quot;write_kilobytes&quot; : 496864
            }
          ],
          &quot;total&quot; : {
            &quot;operations&quot; : 77048,
            &quot;read_operations&quot; : 7736,
            &quot;write_operations&quot; : 69312,
            &quot;read_kilobytes&quot; : 230396,
            &quot;write_kilobytes&quot; : 496864
          }
        }
      }
    },
    &quot;tDet2llLQaydoxfYT4TcpQ&quot; : {
      &quot;timestamp&quot; : 1595231421993,
      &quot;name&quot; : &quot;master-topology-es-1&quot;,
      &quot;transport_address&quot; : &quot;10.244.0.55:9300&quot;,
      &quot;host&quot; : &quot;10.244.0.55&quot;,
      &quot;ip&quot; : &quot;10.244.0.55:9300&quot;,
      &quot;roles&quot; : [
        &quot;master&quot;
      ],
      &quot;fs&quot; : {
        &quot;timestamp&quot; : 1595231422020,
        &quot;total&quot; : {
          &quot;total_in_bytes&quot; : 84054204416,
          &quot;free_in_bytes&quot; : 10231291904,
          &quot;available_in_bytes&quot; : 5917450240
        },
        &quot;data&quot; : [
          {
            &quot;path&quot; : &quot;/usr/share/elasticsearch/data/nodes/0&quot;,
            &quot;mount&quot; : &quot;/usr/share/elasticsearch/data (/dev/sda3)&quot;,
            &quot;type&quot; : &quot;ext4&quot;,
            &quot;total_in_bytes&quot; : 84054204416,
            &quot;free_in_bytes&quot; : 10231291904,
            &quot;available_in_bytes&quot; : 5917450240
          }
        ],
        &quot;io_stats&quot; : {
          &quot;devices&quot; : [
            {
              &quot;device_name&quot; : &quot;sda3&quot;,
              &quot;operations&quot; : 77766,
              &quot;read_operations&quot; : 7748,
              &quot;write_operations&quot; : 70018,
              &quot;read_kilobytes&quot; : 230576,
              &quot;write_kilobytes&quot; : 505892
            }
          ],
          &quot;total&quot; : {
            &quot;operations&quot; : 77766,
            &quot;read_operations&quot; : 7748,
            &quot;write_operations&quot; : 70018,
            &quot;read_kilobytes&quot; : 230576,
            &quot;write_kilobytes&quot; : 505892
          }
        }
      }
    }
  }
}
</code></pre>

Kamol Hasan
<p>So, here is my problem: I've assigned storage of <code>200Mi</code> to each Elasticsearch data node via PVC. When I checked the <code>_nodes/stats</code>:</p>
<pre><code> &quot;total&quot; : {
      &quot;total_in_bytes&quot; : 84054204416, &lt;-- ~84Gi
      &quot;free_in_bytes&quot; : 10231291904,
      &quot;available_in_bytes&quot; : 5917450240
    },
</code></pre>
<p>So, the expected value should be <code>200Mi</code> instead of <code>~84Gi</code>.</p>
<p><strong>Explanation:</strong></p>
<p>Since I'm in <a href="https://github.com/kubernetes-sigs/kind" rel="nofollow noreferrer">kind</a> cluster. The default storage is provisioned by <code>hostpath</code> and we get a directory mounted from the host, not a separate file system.</p>
<p>And <code>kind</code> uses <a href="https://github.com/rancher/local-path-provisioner" rel="nofollow noreferrer">rancher/local-path-provisioner</a>, and the current limitation of using it are:</p>
<pre><code>- No support for the volume capacity limit currently.
    - The capacity limit will be ignored for now.
</code></pre>
<p>That's why I was getting <code>~84Gi</code> (host) instead of <code>200Mi</code> (assigned via PVC).</p>
