{"action":"created","issue":{"url":"https://api.github.com/repos/rook/rook/issues/3539","repository_url":"https://api.github.com/repos/rook/rook","labels_url":"https://api.github.com/repos/rook/rook/issues/3539/labels{/name}","comments_url":"https://api.github.com/repos/rook/rook/issues/3539/comments","events_url":"https://api.github.com/repos/rook/rook/issues/3539/events","html_url":"https://github.com/rook/rook/issues/3539","id":474205937,"node_id":"MDU6SXNzdWU0NzQyMDU5Mzc=","number":3539,"title":"Update bluestore OSDs with new format in v14.2.2","user":{"login":"travisn","id":1048514,"node_id":"MDQ6VXNlcjEwNDg1MTQ=","avatar_url":"https://avatars0.githubusercontent.com/u/1048514?v=4","gravatar_id":"","url":"https://api.github.com/users/travisn","html_url":"https://github.com/travisn","followers_url":"https://api.github.com/users/travisn/followers","following_url":"https://api.github.com/users/travisn/following{/other_user}","gists_url":"https://api.github.com/users/travisn/gists{/gist_id}","starred_url":"https://api.github.com/users/travisn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/travisn/subscriptions","organizations_url":"https://api.github.com/users/travisn/orgs","repos_url":"https://api.github.com/users/travisn/repos","events_url":"https://api.github.com/users/travisn/events{/privacy}","received_events_url":"https://api.github.com/users/travisn/received_events","type":"User","site_admin":false},"labels":[{"id":479456042,"node_id":"MDU6TGFiZWw0Nzk0NTYwNDI=","url":"https://api.github.com/repos/rook/rook/labels/ceph%20-","name":"ceph -","color":"ef5c55","default":false},{"id":1232323300,"node_id":"MDU6TGFiZWwxMjMyMzIzMzAw","url":"https://api.github.com/repos/rook/rook/labels/ceph-osd","name":"ceph-osd","color":"ad0036","default":false},{"id":405241117,"node_id":"MDU6TGFiZWw0MDUyNDExMTc=","url":"https://api.github.com/repos/rook/rook/labels/feature","name":"feature","color":"84b6eb","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2019-07-29T19:00:08Z","updated_at":"2019-08-20T14:14:46Z","closed_at":null,"author_association":"MEMBER","body":"<!-- **Are you in the right place?**\r\n1. For issues or feature requests, please create an issue in this repository.\r\n2. For general technical and non-technical questions, we are happy to help you on our [Rook.io Slack](https://slack.rook.io/).\r\n3. Did you already search the existing open issues for anything similar? -->\r\n\r\n**Is this a bug report or feature request?**\r\n* Feature Request\r\n\r\n**What should the feature do:**\r\nWhen the underlying data format changes in the OSDs, we need to perform a long running operation to update the OSDs. The requirement as seen [here](https://github.com/ceph/ceph/pull/29011/files#diff-662a7de9ea1019d168ac10d9a198c4c9R25) in certain upgrade scenarios is:\r\n- Stop the OSD\r\n- Run the repair on the device: `ceph-bluestore-tool repair --path ...`\r\n- Start the OSD\r\n\r\nThis is specifically seen when updating from luminous or mimic to nautilus v14.2.2. There is a warning that:\r\n```\r\nLegacy BlueStore stats reporting detected on N OSD(s)\r\n```\r\n\r\nMore background can seen in [this BZ](https://bugzilla.redhat.com/show_bug.cgi?id=1729580).\r\n\r\n## Design for updating OSDs\r\nRook will need to automate this process. There isn't a feasible way for the admin to manually upgrade. Consider the following design:\r\n- Rook runs the orchestration, which triggers the \"osd prepare\" job on each node\r\n- The osd prepare job detects if the OSDs are in a state that they need to be repaired for the data path update\r\n- The operator starts the OSDs like usual\r\n- If there are OSDs that require repair, start a goroutine to do the following:\r\n   - For each OSD that requires repair:\r\n      - Take a lock on the OSD orchestration (so the operator doesn't try to update the osd spec in parallel)\r\n      - Stop the osd pod (set its deployment replica count to 0)\r\n      - Run a job that runs the repair command on the OSD\r\n      - When the job completes, start the OSD pod again\r\n      - Release the OSD lock\r\n\r\nThis will generally be a useful pattern going forward for other OSD repairs in the future.\r\n@liewegas Does this seem reasonable or can you think of a simpler repair path? "},"comment":{"url":"https://api.github.com/repos/rook/rook/issues/comments/523034805","html_url":"https://github.com/rook/rook/issues/3539#issuecomment-523034805","issue_url":"https://api.github.com/repos/rook/rook/issues/3539","id":523034805,"node_id":"MDEyOklzc3VlQ29tbWVudDUyMzAzNDgwNQ==","user":{"login":"guits","id":3717838,"node_id":"MDQ6VXNlcjM3MTc4Mzg=","avatar_url":"https://avatars1.githubusercontent.com/u/3717838?v=4","gravatar_id":"","url":"https://api.github.com/users/guits","html_url":"https://github.com/guits","followers_url":"https://api.github.com/users/guits/followers","following_url":"https://api.github.com/users/guits/following{/other_user}","gists_url":"https://api.github.com/users/guits/gists{/gist_id}","starred_url":"https://api.github.com/users/guits/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guits/subscriptions","organizations_url":"https://api.github.com/users/guits/orgs","repos_url":"https://api.github.com/users/guits/repos","events_url":"https://api.github.com/users/guits/events{/privacy}","received_events_url":"https://api.github.com/users/guits/received_events","type":"User","site_admin":false},"created_at":"2019-08-20T14:14:46Z","updated_at":"2019-08-20T14:14:46Z","author_association":"NONE","body":"> This is specifically seen when updating from luminous or mimic to nautilus v14.2.2. There is a warning that:\r\n\r\nFrom what I'm understanding this can even happen when upgrading from nautilus v14.2.0 or v14.2.1 to v14.2.2, right?\r\n"}}
{"actor":{"display_login":"guits"}}