{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/longhorn/issues/60","repository_url":"https://api.github.com/repos/rancher/longhorn","labels_url":"https://api.github.com/repos/rancher/longhorn/issues/60/labels{/name}","comments_url":"https://api.github.com/repos/rancher/longhorn/issues/60/comments","events_url":"https://api.github.com/repos/rancher/longhorn/issues/60/events","html_url":"https://github.com/rancher/longhorn/issues/60","id":320278056,"number":60,"title":"Missing tools and/or documentation how to recover from server/cluster crash","user":{"login":"manwegit","id":5763495,"avatar_url":"https://avatars2.githubusercontent.com/u/5763495?v=4","gravatar_id":"","url":"https://api.github.com/users/manwegit","html_url":"https://github.com/manwegit","followers_url":"https://api.github.com/users/manwegit/followers","following_url":"https://api.github.com/users/manwegit/following{/other_user}","gists_url":"https://api.github.com/users/manwegit/gists{/gist_id}","starred_url":"https://api.github.com/users/manwegit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manwegit/subscriptions","organizations_url":"https://api.github.com/users/manwegit/orgs","repos_url":"https://api.github.com/users/manwegit/repos","events_url":"https://api.github.com/users/manwegit/events{/privacy}","received_events_url":"https://api.github.com/users/manwegit/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-04T13:36:36Z","updated_at":"2018-05-07T10:41:15Z","closed_at":null,"author_association":"NONE","body":"I managed to overload my testing kubernetes cluster and all nodes rebooted at the same time.\r\nNow I have 13 volumes (replicas per node) in Error state and only one working.\r\n\r\nIs there tools or documentation how to recover / rebuild /run fsck on the failed replicas. \r\nI'm still running tests so this was not a catastrophe (and no backups exist) but I'd like to know if there is a way to recover manually at least some of the replicas.\r\n"},"comment":{"url":"https://api.github.com/repos/rancher/longhorn/issues/comments/387027223","html_url":"https://github.com/rancher/longhorn/issues/60#issuecomment-387027223","issue_url":"https://api.github.com/repos/rancher/longhorn/issues/60","id":387027223,"user":{"login":"manwegit","id":5763495,"avatar_url":"https://avatars2.githubusercontent.com/u/5763495?v=4","gravatar_id":"","url":"https://api.github.com/users/manwegit","html_url":"https://github.com/manwegit","followers_url":"https://api.github.com/users/manwegit/followers","following_url":"https://api.github.com/users/manwegit/following{/other_user}","gists_url":"https://api.github.com/users/manwegit/gists{/gist_id}","starred_url":"https://api.github.com/users/manwegit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manwegit/subscriptions","organizations_url":"https://api.github.com/users/manwegit/orgs","repos_url":"https://api.github.com/users/manwegit/repos","events_url":"https://api.github.com/users/manwegit/events{/privacy}","received_events_url":"https://api.github.com/users/manwegit/received_events","type":"User","site_admin":false},"created_at":"2018-05-07T10:41:15Z","updated_at":"2018-05-07T10:41:15Z","author_association":"NONE","body":"Well, at least recombine backup without longhorn code snippet:\r\n```\r\ntruncate -s $(cat volume.cfg |jq -r '.Size') restore.img\r\n\r\ncat backups/backup_backup-PICK.cfg |jq -r '.Blocks| .[]| [.Offset, .BlockChecksum]|@tsv' |perl -anle 'if ($F[1]=~/^(..)(..)/) {print \"dd conv=notrunc if=<(zcat blocks/$1/$2/\".$F[1].\".blk ) of=restore.img seek=$F[0] oflag=seek_bytes\" }' > /tmp/restore-image.sh\r\nbash /tmp/restore-image.sh\r\n```"}}
{"actor":{"display_login":"manwegit"}}