{"action":"created","issue":{"url":"https://api.github.com/repos/lizardfs/lizardfs/issues/705","repository_url":"https://api.github.com/repos/lizardfs/lizardfs","labels_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/705/labels{/name}","comments_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/705/comments","events_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/705/events","html_url":"https://github.com/lizardfs/lizardfs/issues/705","id":327592366,"node_id":"MDU6SXNzdWUzMjc1OTIzNjY=","number":705,"title":"Rebalance/replication caused to fill all the disks","user":{"login":"kubikrobert","id":27781296,"node_id":"MDQ6VXNlcjI3NzgxMjk2","avatar_url":"https://avatars2.githubusercontent.com/u/27781296?v=4","gravatar_id":"","url":"https://api.github.com/users/kubikrobert","html_url":"https://github.com/kubikrobert","followers_url":"https://api.github.com/users/kubikrobert/followers","following_url":"https://api.github.com/users/kubikrobert/following{/other_user}","gists_url":"https://api.github.com/users/kubikrobert/gists{/gist_id}","starred_url":"https://api.github.com/users/kubikrobert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubikrobert/subscriptions","organizations_url":"https://api.github.com/users/kubikrobert/orgs","repos_url":"https://api.github.com/users/kubikrobert/repos","events_url":"https://api.github.com/users/kubikrobert/events{/privacy}","received_events_url":"https://api.github.com/users/kubikrobert/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2018-05-30T06:39:27Z","updated_at":"2018-06-02T09:44:06Z","closed_at":null,"author_association":"NONE","body":"Hi again :)\r\n\r\nMy cluster contains 2 goals (ssd and hdd, both with replication):\r\n![obraz](https://user-images.githubusercontent.com/27781296/40703092-6790784e-63e4-11e8-83fc-5a4ac6584775.png)\r\n\r\nFew days ago, I've removed and recreated one of the chunkservers and in the same time - added one additional disk to each chunkserver. Before that, whole cluster had around 77% of HDD goal/disks usage.\r\n\r\nThe removal/recreation caused rebalance to make all the endangered chunks save, here is what I saw todays morning:\r\n\r\n![obraz](https://user-images.githubusercontent.com/27781296/40703191-ab621ffa-63e4-11e8-98f6-0f0a2d131f32.png)\r\n![obraz](https://user-images.githubusercontent.com/27781296/40703207-be31641a-63e4-11e8-81ce-4edc5ae6aae6.png)\r\n\r\nAs you see, there is a huge difference between chunks amount on particular servers (usually 30 million per 2TB disk). The recreated CS has 10 million and is almost full. Why? :(\r\n\r\nThere is still a lot of chunks to make safe/replicate:\r\n![obraz](https://user-images.githubusercontent.com/27781296/40703247-d9cffe8e-63e4-11e8-9e31-8741d1aa3661.png)\r\n![obraz](https://user-images.githubusercontent.com/27781296/40703257-e4fb97aa-63e4-11e8-8e33-f570718db07e.png)\r\n\r\nI'm not sure what just happened.. :(\r\n\r\nIn the master server logs I see a lot of that entries:\r\n\r\n> May 28 14:13:43 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 00000000003B2F43 replication status: Unknown LizardFS error\r\n> May 28 14:13:43 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000001D03C4D replication status: Unknown LizardFS error\r\n> May 28 14:13:43 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000001E408DF replication status: Unknown LizardFS error\r\n> May 28 14:13:43 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 00000000040BBAA0 replication status: Unknown LizardFS error\r\n> May 28 14:13:52 lfs-master03 mfsmaster[26330]: chunk hasn't been deleted since previous loop - retry\r\n> May 28 14:13:53 lfs-master03 mfsmaster[26330]: (10.125.6.19:9422) chunk: 0000000001837246 deletion status: No such chunk\r\n> May 28 20:47:10 lfs-master03 mfsmaster[26330]: chunk hasn't been deleted since previous loop - retry\r\n> May 28 20:47:15 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000002CF78E6 replication status: Unknown LizardFS error\r\n> May 28 20:47:15 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 00000000002AC325 replication status: Unknown LizardFS error\r\n> May 28 20:47:15 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 00000000032F78E6 replication status: Unknown LizardFS error\r\n> May 28 20:47:15 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000003E6C206 replication status: Unknown LizardFS error\r\n> May 28 20:47:15 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000004628B5F replication status: Unknown LizardFS error\r\n> May 28 20:47:26 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 000000000291DDF8 replication status: Unknown LizardFS error\r\n> May 28 20:47:27 lfs-master03 mfsmaster[26330]: (10.125.6.18:9422) chunk: 00000000039F8EF1 deletion status: No such chunk\r\n> May 28 20:47:43 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000002F4159C replication status: Unknown LizardFS error\r\n> May 28 20:55:34 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000002C9BAEC replication status: Unknown LizardFS error\r\n> May 28 20:55:34 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 0000000002E9BAEC replication status: Unknown LizardFS error\r\n> May 28 20:55:34 lfs-master03 mfsmaster[26330]: (10.125.6.21:9422) chunk: 000000000055F047 replication status: Unknown LizardFS error\r\n\r\nShould I set the goal of all files to some single one (without replication) and then back to dual?\r\n\r\nThanks in advance for any help.\r\nRegards"},"comment":{"url":"https://api.github.com/repos/lizardfs/lizardfs/issues/comments/394074208","html_url":"https://github.com/lizardfs/lizardfs/issues/705#issuecomment-394074208","issue_url":"https://api.github.com/repos/lizardfs/lizardfs/issues/705","id":394074208,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NDA3NDIwOA==","user":{"login":"guestisp","id":3944655,"node_id":"MDQ6VXNlcjM5NDQ2NTU=","avatar_url":"https://avatars3.githubusercontent.com/u/3944655?v=4","gravatar_id":"","url":"https://api.github.com/users/guestisp","html_url":"https://github.com/guestisp","followers_url":"https://api.github.com/users/guestisp/followers","following_url":"https://api.github.com/users/guestisp/following{/other_user}","gists_url":"https://api.github.com/users/guestisp/gists{/gist_id}","starred_url":"https://api.github.com/users/guestisp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guestisp/subscriptions","organizations_url":"https://api.github.com/users/guestisp/orgs","repos_url":"https://api.github.com/users/guestisp/repos","events_url":"https://api.github.com/users/guestisp/events{/privacy}","received_events_url":"https://api.github.com/users/guestisp/received_events","type":"User","site_admin":false},"created_at":"2018-06-02T09:44:06Z","updated_at":"2018-06-02T09:44:06Z","author_association":"NONE","body":"The only way to share a filesystem is to use a clustered filesystem.\r\n\r\nExt*, xfs, ZFS, and so on can't bè shared"}}
{"actor":{"display_login":"guestisp"}}