{"action":"opened","issue":{"url":"https://api.github.com/repos/rook/rook/issues/1601","repository_url":"https://api.github.com/repos/rook/rook","labels_url":"https://api.github.com/repos/rook/rook/issues/1601/labels{/name}","comments_url":"https://api.github.com/repos/rook/rook/issues/1601/comments","events_url":"https://api.github.com/repos/rook/rook/issues/1601/events","html_url":"https://github.com/rook/rook/issues/1601","id":308194556,"number":1601,"title":"Docker Upgrade Causes OSD Failure","user":{"login":"patrickstjohn","id":7337041,"avatar_url":"https://avatars1.githubusercontent.com/u/7337041?v=4","gravatar_id":"","url":"https://api.github.com/users/patrickstjohn","html_url":"https://github.com/patrickstjohn","followers_url":"https://api.github.com/users/patrickstjohn/followers","following_url":"https://api.github.com/users/patrickstjohn/following{/other_user}","gists_url":"https://api.github.com/users/patrickstjohn/gists{/gist_id}","starred_url":"https://api.github.com/users/patrickstjohn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/patrickstjohn/subscriptions","organizations_url":"https://api.github.com/users/patrickstjohn/orgs","repos_url":"https://api.github.com/users/patrickstjohn/repos","events_url":"https://api.github.com/users/patrickstjohn/events{/privacy}","received_events_url":"https://api.github.com/users/patrickstjohn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-23T22:18:00Z","updated_at":"2018-03-23T22:18:00Z","closed_at":null,"author_association":"NONE","body":"* Bug Report\r\n\r\n**Bug Report**\r\nUpgrade of docker in a bare-metal deployment (Kubernetes 1.9.5, Rook 0.7.1, RHEL 7.4).\r\n\r\nDuring an RPM upgrade of Docker 17.03 to 17.09.1 the Ceph cluster became unhealthy with the following errors on many OSD's. The pods would not come out of this error state until the directory was wiped clean. Unfortunately, this continued in a cascading fashion with all the OSD pods eventually falling into this failure state.\r\n\r\n2018-03-23 21:53:28.008925 I | exec: Running command: ceph-osd --foreground --id=72 --cluster=rook --osd-data=/var/lib/rook/osd72 --conf=/var/lib/rook/osd72/rook.config --keyring=/var/lib/rook/osd72/keyring --osd-uuid=6266abb6-540b-4ba2-85c3-4559b8d3957b\r\n2018-03-23 21:53:28.009893 I | proc: retry (total 323). started process [ceph-osd --foreground --id=72 --cluster=rook --osd-data=/var/lib/rook/osd72 --conf=/var/lib/rook/osd72/rook.config --keyring=/var/lib/rook/osd72/keyring --osd-uuid=6266abb6-540b-4ba2-85c3-4559b8d3957b]\r\n2018-03-23 21:53:28.026917 I | osd72: 2018-03-23 21:53:28.026801 7fcafa3afe00  0 ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable), process (unknown), pid 5685\r\n2018-03-23 21:53:28.028246 I | osd72: starting osd.72 at - osd_data /var/lib/rook/osd72 /var/lib/ceph/osd/rook-72/journal\r\n2018-03-23 21:53:28.029243 I | osd72: 2018-03-23 21:53:28.029226 7fcafa3afe00  0 pidfile_write: ignore empty --pid-file\r\n2018-03-23 21:53:28.034251 I | osd72: 2018-03-23 21:53:28.034225 7fcafa3afe00  0 load: jerasure load: lrc load: isa \r\n2018-03-23 21:53:28.034378 I | osd72: 2018-03-23 21:53:28.034278 7fcafa3afe00 -1 bluestore(/var/lib/rook/osd72) _lock_fsid failed to lock /var/lib/rook/osd72/fsid (is another ceph-osd still running?)(11) Resource temporarily unavailable\r\n2018-03-23 21:53:28.034546 I | osd72: 2018-03-23 21:53:28.034519 7fcafa3afe00 -1 bluestore(/var/lib/rook/osd72) _lock_fsid failed to lock /var/lib/rook/osd72/fsid (is another ceph-osd still running?)(11) Resource temporarily unavailable\r\n2018-03-23 21:53:28.034553 I | osd72: 2018-03-23 21:53:28.034527 7fcafa3afe00 -1 osd.72 0 OSD::pre_init: object store '/var/lib/rook/osd72' is currently in use. (Is ceph-osd already running?)\r\n2018-03-23 21:53:28.034558 I | osd72: 2018-03-23 21:53:28.034528 7fcafa3afe00 -1  ** ERROR: osd pre_init failed: (16) Device or resource busy\r\n2018-03-23 21:53:28.038601 I | osd72: 2018-03-23 21:53:28.034278 7fcafa3afe00 -1 bluestore(/var/lib/rook/osd72) _lock_fsid failed to lock /var/lib/rook/osd72/fsid (is another ceph-osd still running?)(11) Resource temporarily unavailable\r\n2018-03-23 21:53:28.038606 I | osd72: 2018-03-23 21:53:28.034519 7fcafa3afe00 -1 bluestore(/var/lib/rook/osd72) _lock_fsid failed to lock /var/lib/rook/osd72/fsid (is another ceph-osd still running?)(11) Resource temporarily unavailable\r\n2018-03-23 21:53:28.038608 I | osd72: 2018-03-23 21:53:28.034527 7fcafa3afe00 -1 osd.72 0 OSD::pre_init: object store '/var/lib/rook/osd72' is currently in use. (Is ceph-osd already running?)\r\n2018-03-23 21:53:28.038611 I | osd72: 2018-03-23 21:53:28.034528 7fcafa3afe00 -1  ** ERROR: osd pre_init failed: (16) Device or resource busy\r\n2018-03-23 21:53:28.038842 I | proc: process 5685 completed.  Exited: true, ExitStatus: 1, Signaled: false, Signal: -1, &{Path:/usr/bin/ceph-osd Args:[ceph-osd --foreground --id=72 --cluster=rook --osd-data=/var/lib/rook/osd72 --conf=/var/lib/rook/osd72/rook.config --keyring=/var/lib/rook/osd72/keyring --osd-uuid=6266abb6-540b-4ba2-85c3-4559b8d3957b] Env:[] Dir: Stdin:<nil> Stdout:0xc42000c030 Stderr:0xc42000c040 ExtraFiles:[] SysProcAttr:<nil> Process:0xc420668270 ProcessState:<nil> ctx:<nil> lookPathErr:<nil> finished:false childFiles:[0xc42000c048 0xc42000c030 0xc42000c040] closeAfterStart:[0xc42000c030 0xc42000c040 0xc42000c048] closeAfterWait:[0xc42000c028 0xc42000c038] goroutine:[] errch:0xc4203ac060 waitDone:<nil>}\r\n2018-03-23 21:53:28.038867 I | proc: starting process [ceph-osd --foreground --id=72 --cluster=rook --osd-data=/var/lib/rook/osd72 --conf=/var/lib/rook/osd72/rook.config --keyring=/var/lib/rook/osd72/keyring --osd-uuid=6266abb6-540b-4ba2-85c3-4559b8d3957b] again after 30.0 seconds\r\n\r\n* Steps to Reproduce\r\n1. Upgrade Docker on a Rook deployment (1 node at a time)\r\n2. Restart Docker Service\r\n3. Watch OSDs degrade and eventually fail throughout the cluster\r\n\r\n**Environment**:\r\n* OS (e.g. from /etc/os-release): RHEL 7.4\r\n* Kernel (e.g. `uname -a`): Linux 4.4.115-1.el7.elrepo.x86_64 #1 SMP Sat Feb 3 20:11:41 EST 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n* Rook version (use `rook version` inside of a Rook Pod): 0.7.1\r\n* Kubernetes version (use `kubectl version`): 1.9.5\r\n* Ceph status (use `ceph health` in the [Rook toolbox](https://Rook.io/docs/Rook/master/toolbox.html)):\r\nHEALTH_WARN 3 osds down; 3 hosts (3 osds) down; 161/45133444 objects unfound (0.000%); Reduced data availability: 718 pgs inactive, 55 pgs down, 106 pgs peering, 920 pgs incomplete, 5 pgs stale; Degraded data redundancy: 1547362/111006109 objects degraded (1.394%), 2219 pgs degraded, 243 pgs undersized; 164 slow requests are blocked > 32 sec\r\n\r\nI think when we try this again we'll likely do a single node at a time and wait for the cluster to become healthy. Any suggestions on better upgrade patterns for key components would be helpful in documentation (unless I missed this somewhere which is possible).\r\n"}}
{"actor":{"display_login":"patrickstjohn"}}