<p>Having an issues with glusterfs. Looks like not all bricks are using by PODs.</p>

<p>So I'm using Kubernetes v1.11, Heketi v9.0.0 and glusterfs 4.1.5</p>

<p>Previously I had strange issues with one worker node so I need to reboot this node. After this I've got a problems with heketi pod. Looks like it couldn't mount heketi db. 
I fixed this issue, it was able to start, but when I checked gluster mounted share in PODs, I've noticed that I only have the latest data there.<br>
First of all I've checked all peers status:</p>

<pre><code>Number of Peers: 2

Hostname: 192.168.2.148
Uuid: a5b95e50-5fba-41a4-ad4d-e0c0f32686e9
State: Peer in Cluster (Connected)

Hostname: 192.168.2.70
Uuid: 0d91679f-bd49-4cd2-b003-383d8208f81b
State: Peer in Cluster (Connected)
</code></pre>

<p>then compared all the bricks inside glusterfs volumes using </p>

<pre><code>gluster volume info
</code></pre>

<p>for all gluster nodes and volumes with the heketi topology and all is fine there.<br>
Using <code>lvdisplay</code> on my glusterfs nodes I checked path for all bricks and even mounted some of them on host node to ensure that the data is still available inside them - found old data inside.</p>

<p>some output from glusterfs instance:</p>

<pre><code>gluster volume list
heketidbstorage
vol_3e49f0d33f5610cae6808cc77e028698
vol_d592d9bed635ad3f18b32fad15b30e5e

# gluster volume info vol_3e49f0d33f5610cae6808cc77e028698

Volume Name: vol_3e49f0d33f5610cae6808cc77e028698
Type: Distributed-Replicate
Volume ID: ca82ebcd-5e8a-4969-b7a5-c17b7e9b7b9e
Status: Started
Snapshot Count: 0
Number of Bricks: 5 x 3 = 15
Transport-type: tcp
Bricks:
Brick1: 192.168.2.96:/var/lib/heketi/mounts/vg_0c8b46493bec60ea4531d7efbc5160b3/brick_e1879c9d08c2da4691ae0c3e85b3d090/brick
Brick2: 192.168.2.148:/var/lib/heketi/mounts/vg_0e3d99f43455efbaad08a9835e5829b5/brick_c0074cec63bd11946ee22d981346d76a/brick
Brick3: 192.168.2.70:/var/lib/heketi/mounts/vg_c974ac16bc0c783c55a29f83eeb71962/brick_b7a72f169526b8c154784ebe0611f4c0/brick
Brick4: 192.168.2.70:/var/lib/heketi/mounts/vg_c974ac16bc0c783c55a29f83eeb71962/brick_5044931511b6f0c9bc28e3305a12de34/brick
Brick5: 192.168.2.148:/var/lib/heketi/mounts/vg_0e3d99f43455efbaad08a9835e5829b5/brick_8fbb59b27b61117eb5b87873e7371d56/brick
Brick6: 192.168.2.96:/var/lib/heketi/mounts/vg_0c8b46493bec60ea4531d7efbc5160b3/brick_f89c254d5b8905380e3c3d1cc5ca22ca/brick
Brick7: 192.168.2.148:/var/lib/heketi/mounts/vg_0e3d99f43455efbaad08a9835e5829b5/brick_d5e3a284457d35d2245b8a93f4a700aa/brick
Brick8: 192.168.2.70:/var/lib/heketi/mounts/vg_c974ac16bc0c783c55a29f83eeb71962/brick_397fc0cc8465f9517ea859af25f928db/brick
Brick9: 192.168.2.96:/var/lib/heketi/mounts/vg_0c8b46493bec60ea4531d7efbc5160b3/brick_a69906bc8f4662288f7578c6770660fc/brick
Brick10: 192.168.2.96:/var/lib/heketi/mounts/vg_0c8b46493bec60ea4531d7efbc5160b3/brick_03e6de08eeb075dde0943c7e0191ca3e/brick
Brick11: 192.168.2.148:/var/lib/heketi/mounts/vg_0e3d99f43455efbaad08a9835e5829b5/brick_d762e5bf6013e3bd31e88e02ce9f06c0/brick
Brick12: 192.168.2.70:/var/lib/heketi/mounts/vg_c974ac16bc0c783c55a29f83eeb71962/brick_aca4a7b0d51ca95376ec7f29515d290f/brick
Brick13: 192.168.2.148:/var/lib/heketi/mounts/vg_0e3d99f43455efbaad08a9835e5829b5/brick_231f645020d2a0c691173cee432ace9e/brick
Brick14: 192.168.2.96:/var/lib/heketi/mounts/vg_0c8b46493bec60ea4531d7efbc5160b3/brick_807002a90ae315d40138a6031096d812/brick
Brick15: 192.168.2.70:/var/lib/heketi/mounts/vg_c974ac16bc0c783c55a29f83eeb71962/brick_f516b9811f1581d5fc453692e2a15183/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

# gluster volume status vol_3e49f0d33f5610cae6808cc77e028698 detail

Status of volume: vol_3e49f0d33f5610cae6808cc77e028698
------------------------------------------------------------------------------
Brick                : Brick 192.168.2.96:/var/lib/heketi/mounts/vg_0c8b46493bec60ea4531d7efbc5160b3/brick_e1879c9d08c2da4691ae0c3e85b3d090/brick
TCP Port             : 49153
RDMA Port            : 0
Online               : Y
Pid                  : 187
File System          : xfs
Device               : /dev/mapper/vg_0c8b46493bec60ea4531d7efbc5160b3-brick_e1879c9d08c2da4691ae0c3e85b3d090
Mount Options        : rw,noatime,nouuid,attr2,inode64,logbsize=128k,sunit=256,swidth=512,noquota
Inode Size           : 512
Disk Space Free      : 566.3GB
Total Disk Space     : 1.4TB
Inode Count          : 155713088
Free Inodes          : 155419488
</code></pre>

<p>When I checked gluster mounted share using df -h inside POD I see:</p>

<pre><code>192.168.2.148:vol_3e49f0d33f5610cae6808cc77e028698  486G  304G  183G  63% /var/lib/kubelet/pods/6804de60-97e8-11e9-be43-12eea8244508/volumes/kubernetes.io~glusterfs/pvc-d9222fa3-b649-11e8-9583-12eea8244508
</code></pre>

<p>but should be few Terrabytes.</p>

Oles Rid
<p>Finally I've found a solution. It was related to shared-brick-count known issue with glusterfs v4.
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1517260" rel="nofollow noreferrer">Bug 1517260 - Volume wrong size </a></p>
