{"action":"created","issue":{"url":"https://api.github.com/repos/lizardfs/lizardfs/issues/705","repository_url":"https://api.github.com/repos/lizardfs/lizardfs","labels_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/705/labels{/name}","comments_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/705/comments","events_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/705/events","html_url":"https://github.com/lizardfs/lizardfs/issues/705","id":327592366,"node_id":"MDU6SXNzdWUzMjc1OTIzNjY=","number":705,"title":"Rebalance/replication caused to fill all the disks","user":{"login":"kubikrobert","id":27781296,"node_id":"MDQ6VXNlcjI3NzgxMjk2","avatar_url":"https://avatars2.githubusercontent.com/u/27781296?v=4","gravatar_id":"","url":"https://api.github.com/users/kubikrobert","html_url":"https://github.com/kubikrobert","followers_url":"https://api.github.com/users/kubikrobert/followers","following_url":"https://api.github.com/users/kubikrobert/following{/other_user}","gists_url":"https://api.github.com/users/kubikrobert/gists{/gist_id}","starred_url":"https://api.github.com/users/kubikrobert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubikrobert/subscriptions","organizations_url":"https://api.github.com/users/kubikrobert/orgs","repos_url":"https://api.github.com/users/kubikrobert/repos","events_url":"https://api.github.com/users/kubikrobert/events{/privacy}","received_events_url":"https://api.github.com/users/kubikrobert/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-05-30T06:39:27Z","updated_at":"2018-06-01T11:58:17Z","closed_at":null,"author_association":"NONE","body":"Hi again :)\r\n\r\nMy cluster contains 2 goals (ssd and hdd, both with replication):\r\n![obraz](https://user-images.githubusercontent.com/27781296/40703092-6790784e-63e4-11e8-83fc-5a4ac6584775.png)\r\n\r\nFew days ago, I've removed and recreated one of the chunkservers and in the same time - added one additional disk to each chunkserver. Before that, whole cluster had around 77% of HDD goal/disks usage.\r\n\r\nThe removal/recreation caused rebalance to make all the endangered chunks save, here is what I saw todays morning:\r\n\r\n![obraz](https://user-images.githubusercontent.com/27781296/40703191-ab621ffa-63e4-11e8-98f6-0f0a2d131f32.png)\r\n![obraz](https://user-images.githubusercontent.com/27781296/40703207-be31641a-63e4-11e8-81ce-4edc5ae6aae6.png)\r\n\r\nAs you see, there is a huge difference between chunks amount on particular servers (usually 30 million per 2TB disk). The recreated CS has 10 million and is almost full. Why? :(\r\n\r\nThere is still a lot of chunks to make safe/replicate:\r\n![obraz](https://user-images.githubusercontent.com/27781296/40703247-d9cffe8e-63e4-11e8-9e31-8741d1aa3661.png)\r\n![obraz](https://user-images.githubusercontent.com/27781296/40703257-e4fb97aa-63e4-11e8-8e33-f570718db07e.png)\r\n\r\nI'm not sure what just happened.. :(\r\n\r\nIn the master server logs I see a lot of that entries:\r\n\r\n> May 28 14:13:43 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 00000000003B2F43 replication status: Unknown LizardFS error\r\n> May 28 14:13:43 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000001D03C4D replication status: Unknown LizardFS error\r\n> May 28 14:13:43 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000001E408DF replication status: Unknown LizardFS error\r\n> May 28 14:13:43 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 00000000040BBAA0 replication status: Unknown LizardFS error\r\n> May 28 14:13:52 lfs-master03 mfsmaster[26330]: chunk hasn't been deleted since previous loop - retry\r\n> May 28 14:13:53 lfs-master03 mfsmaster[26330]: (10.125.6.19:9422) chunk: 0000000001837246 deletion status: No such chunk\r\n> May 28 20:47:10 lfs-master03 mfsmaster[26330]: chunk hasn't been deleted since previous loop - retry\r\n> May 28 20:47:15 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000002CF78E6 replication status: Unknown LizardFS error\r\n> May 28 20:47:15 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 00000000002AC325 replication status: Unknown LizardFS error\r\n> May 28 20:47:15 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 00000000032F78E6 replication status: Unknown LizardFS error\r\n> May 28 20:47:15 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000003E6C206 replication status: Unknown LizardFS error\r\n> May 28 20:47:15 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000004628B5F replication status: Unknown LizardFS error\r\n> May 28 20:47:26 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 000000000291DDF8 replication status: Unknown LizardFS error\r\n> May 28 20:47:27 lfs-master03 mfsmaster[26330]: (10.125.6.18:9422) chunk: 00000000039F8EF1 deletion status: No such chunk\r\n> May 28 20:47:43 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000002F4159C replication status: Unknown LizardFS error\r\n> May 28 20:55:34 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000002C9BAEC replication status: Unknown LizardFS error\r\n> May 28 20:55:34 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000002E9BAEC replication status: Unknown LizardFS error\r\n> May 28 20:55:34 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 000000000055F047 replication status: Unknown LizardFS error\r\n\r\nShould I set the goal of all files to some single one (without replication) and then back to dual?\r\n\r\nThanks in advance for any help.\r\nRegards"},"comment":{"url":"https://api.github.com/repos/lizardfs/lizardfs/issues/comments/393859020","html_url":"https://github.com/lizardfs/lizardfs/issues/705#issuecomment-393859020","issue_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/705","id":393859020,"node_id":"MDEyOklzc3VlQ29tbWVudDM5Mzg1OTAyMA==","user":{"login":"pkali","id":23016428,"node_id":"MDQ6VXNlcjIzMDE2NDI4","avatar_url":"https://avatars3.githubusercontent.com/u/23016428?v=4","gravatar_id":"","url":"https://api.github.com/users/pkali","html_url":"https://github.com/pkali","followers_url":"https://api.github.com/users/pkali/followers","following_url":"https://api.github.com/users/pkali/following{/other_user}","gists_url":"https://api.github.com/users/pkali/gists{/gist_id}","starred_url":"https://api.github.com/users/pkali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pkali/subscriptions","organizations_url":"https://api.github.com/users/pkali/orgs","repos_url":"https://api.github.com/users/pkali/repos","events_url":"https://api.github.com/users/pkali/events{/privacy}","received_events_url":"https://api.github.com/users/pkali/received_events","type":"User","site_admin":false},"created_at":"2018-06-01T11:58:17Z","updated_at":"2018-06-01T11:58:17Z","author_association":"NONE","body":"HDD_PUNCH_HOLES - no, it works only when a chunk is written, so not an easy situation when there is no space left.\r\nWe have checked deeper and unfortunately holes are being punched in 64KiB multiplies only, so it will not help here, I'm afraid.\r\nStill, creating loopback FS is a solution - you can create the one that grows in size (sparse-file device) and move there files one by one. \r\nWe'll look again at the question why this started to show up after a disk replacement. My initial guess is that it is a kind of a coincidence - LizardFS is replicating and rebalancing lazily and after set periods of time, so it might happen with the old HDD as well."}}
{"actor":{"display_login":"pkali"}}