{"action":"created","issue":{"url":"https://api.github.com/repos/sapcc/swift-drive-autopilot/issues/10","repository_url":"https://api.github.com/repos/sapcc/swift-drive-autopilot","labels_url":"https://api.github.com/repos/sapcc/swift-drive-autopilot/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/sapcc/swift-drive-autopilot/issues/10/comments","events_url":"https://api.github.com/repos/sapcc/swift-drive-autopilot/issues/10/events","html_url":"https://github.com/sapcc/swift-drive-autopilot/issues/10","id":191994508,"number":10,"title":"all devices being flagged as broken after restart","user":{"login":"majewsky","id":24696,"avatar_url":"https://avatars.githubusercontent.com/u/24696?v=3","gravatar_id":"","url":"https://api.github.com/users/majewsky","html_url":"https://github.com/majewsky","followers_url":"https://api.github.com/users/majewsky/followers","following_url":"https://api.github.com/users/majewsky/following{/other_user}","gists_url":"https://api.github.com/users/majewsky/gists{/gist_id}","starred_url":"https://api.github.com/users/majewsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/majewsky/subscriptions","organizations_url":"https://api.github.com/users/majewsky/orgs","repos_url":"https://api.github.com/users/majewsky/repos","events_url":"https://api.github.com/users/majewsky/events{/privacy}","received_events_url":"https://api.github.com/users/majewsky/received_events","type":"User","site_admin":false},"labels":[{"id":460709898,"url":"https://api.github.com/repos/sapcc/swift-drive-autopilot/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":{"login":"majewsky","id":24696,"avatar_url":"https://avatars.githubusercontent.com/u/24696?v=3","gravatar_id":"","url":"https://api.github.com/users/majewsky","html_url":"https://github.com/majewsky","followers_url":"https://api.github.com/users/majewsky/followers","following_url":"https://api.github.com/users/majewsky/following{/other_user}","gists_url":"https://api.github.com/users/majewsky/gists{/gist_id}","starred_url":"https://api.github.com/users/majewsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/majewsky/subscriptions","organizations_url":"https://api.github.com/users/majewsky/orgs","repos_url":"https://api.github.com/users/majewsky/repos","events_url":"https://api.github.com/users/majewsky/events{/privacy}","received_events_url":"https://api.github.com/users/majewsky/received_events","type":"User","site_admin":false},"assignees":[{"login":"majewsky","id":24696,"avatar_url":"https://avatars.githubusercontent.com/u/24696?v=3","gravatar_id":"","url":"https://api.github.com/users/majewsky","html_url":"https://github.com/majewsky","followers_url":"https://api.github.com/users/majewsky/followers","following_url":"https://api.github.com/users/majewsky/following{/other_user}","gists_url":"https://api.github.com/users/majewsky/gists{/gist_id}","starred_url":"https://api.github.com/users/majewsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/majewsky/subscriptions","organizations_url":"https://api.github.com/users/majewsky/orgs","repos_url":"https://api.github.com/users/majewsky/repos","events_url":"https://api.github.com/users/majewsky/events{/privacy}","received_events_url":"https://api.github.com/users/majewsky/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-11-28T12:31:23Z","updated_at":"2016-11-28T14:15:37Z","closed_at":null,"body":"Upon restarting the autopilot in our staging environment, I encountered the following \"interesting\" behavior:\r\n\r\n```\r\n2016/11/28 12:22:13 INFO: event received: new device found: /dev/sdc\r\n2016/11/28 12:22:13 INFO: discovered /dev/sdc to be mapped to /dev/mapper/81dde30aa3d9bf946fb0281f260b68e4 already\r\n2016/11/28 12:22:13 INFO: discovered /dev/mapper/81dde30aa3d9bf946fb0281f260b68e4 to be mounted at /srv/node/swift-01 already\r\n2016/11/28 12:22:13 INFO: event received: new device found: /dev/sdd\r\n2016/11/28 12:22:13 INFO: discovered /dev/sdd to be mapped to /dev/mapper/dbc0b33565fc9c0784c8ed66ec7d123e already\r\n2016/11/28 12:22:13 INFO: discovered /dev/mapper/dbc0b33565fc9c0784c8ed66ec7d123e to be mounted at /srv/node/swift-03 already\r\n2016/11/28 12:22:13 INFO: event received: new device found: /dev/sde\r\n2016/11/28 12:22:13 INFO: discovered /dev/sde to be mapped to /dev/mapper/c338434f254439c7cd0c8ba21a872405 already\r\n2016/11/28 12:22:13 INFO: discovered /dev/mapper/c338434f254439c7cd0c8ba21a872405 to be mounted at /srv/node/swift-02 already\r\n2016/11/28 12:22:43 INFO: event received: scheduled consistency check\r\n2016/11/28 12:22:43 ERROR: expected /dev/mapper/81dde30aa3d9bf946fb0281f260b68e4 to be mounted at /srv/node/swift-01, but is not mounted anymore\r\n2016/11/28 12:22:43 INFO: flagging /dev/sdc as broken because of previous error\r\n2016/11/28 12:22:43 INFO: To reinstate this drive into the cluster, delete the symlink at /run/swift-storage/broken/81dde30aa3d9bf946fb0281f260b68e4\r\n2016/11/28 12:22:43 ERROR: expected /dev/mapper/dbc0b33565fc9c0784c8ed66ec7d123e to be mounted at /srv/node/swift-03, but is not mounted anymore\r\n2016/11/28 12:22:43 INFO: flagging /dev/sdd as broken because of previous error\r\n2016/11/28 12:22:43 INFO: To reinstate this drive into the cluster, delete the symlink at /run/swift-storage/broken/dbc0b33565fc9c0784c8ed66ec7d123e\r\n2016/11/28 12:22:43 ERROR: expected /dev/mapper/c338434f254439c7cd0c8ba21a872405 to be mounted at /srv/node/swift-02, but is not mounted anymore\r\n2016/11/28 12:22:43 INFO: flagging /dev/sde as broken because of previous error\r\n2016/11/28 12:22:43 INFO: To reinstate this drive into the cluster, delete the symlink at /run/swift-storage/broken/c338434f254439c7cd0c8ba21a872405\r\n2016/11/28 12:23:13 INFO: event received: scheduled consistency check\r\n```"},"comment":{"url":"https://api.github.com/repos/sapcc/swift-drive-autopilot/issues/comments/263281341","html_url":"https://github.com/sapcc/swift-drive-autopilot/issues/10#issuecomment-263281341","issue_url":"https://api.github.com/repos/sapcc/swift-drive-autopilot/issues/10","id":263281341,"user":{"login":"majewsky","id":24696,"avatar_url":"https://avatars.githubusercontent.com/u/24696?v=3","gravatar_id":"","url":"https://api.github.com/users/majewsky","html_url":"https://github.com/majewsky","followers_url":"https://api.github.com/users/majewsky/followers","following_url":"https://api.github.com/users/majewsky/following{/other_user}","gists_url":"https://api.github.com/users/majewsky/gists{/gist_id}","starred_url":"https://api.github.com/users/majewsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/majewsky/subscriptions","organizations_url":"https://api.github.com/users/majewsky/orgs","repos_url":"https://api.github.com/users/majewsky/repos","events_url":"https://api.github.com/users/majewsky/events{/privacy}","received_events_url":"https://api.github.com/users/majewsky/received_events","type":"User","site_admin":false},"created_at":"2016-11-28T14:15:37Z","updated_at":"2016-11-28T14:15:37Z","body":"Ah, it happened again just as I deployed the version with additional debug output onto the cluster.\r\n\r\n```\r\n2016/11/28 14:11:58 DEBUG: chown /var/cache/swift to 1000:1000\r\n2016/11/28 14:11:58 DEBUG: ActiveLUKSMappings = map[string]string{\"/dev/sdc\":\"81dde30aa3d9bf946fb0281f260b68e4\", \"/dev/sde\":\"c338434f254439c7cd0c8ba21a872405\", \"/dev/sdd\":\"dbc0b33565fc9c0784c8ed66ec7d123e\"}\r\n2016/11/28 14:11:58 INFO: event received: new device found: /dev/disk/by-path/ip-10.44.12.251:3260-iscsi-iqn.1992-08.com.netapp:sn.d45559e6e6a511e5a21e000e1e2a7d30:vs.15-lun-0 -> /dev/sdc\r\n2016/11/28 14:11:58 INFO: discovered /dev/sdc to be mapped to /dev/mapper/81dde30aa3d9bf946fb0281f260b68e4 already\r\n2016/11/28 14:11:58 INFO: mounted /dev/mapper/81dde30aa3d9bf946fb0281f260b68e4 to /run/swift-storage/81dde30aa3d9bf946fb0281f260b68e4\r\n2016/11/28 14:11:58 INFO: event received: new device found: /dev/disk/by-path/ip-10.44.12.251:3260-iscsi-iqn.1992-08.com.netapp:sn.d45559e6e6a511e5a21e000e1e2a7d30:vs.15-lun-1 -> /dev/sdd\r\n2016/11/28 14:11:58 INFO: discovered /dev/sdd to be mapped to /dev/mapper/dbc0b33565fc9c0784c8ed66ec7d123e already\r\n2016/11/28 14:11:59 INFO: mounted /dev/mapper/dbc0b33565fc9c0784c8ed66ec7d123e to /run/swift-storage/dbc0b33565fc9c0784c8ed66ec7d123e\r\n2016/11/28 14:11:59 INFO: event received: new device found: /dev/disk/by-path/ip-10.44.12.251:3260-iscsi-iqn.1992-08.com.netapp:sn.d45559e6e6a511e5a21e000e1e2a7d30:vs.15-lun-2 -> /dev/sde\r\n2016/11/28 14:11:59 INFO: discovered /dev/sde to be mapped to /dev/mapper/c338434f254439c7cd0c8ba21a872405 already\r\n2016/11/28 14:11:59 INFO: mounted /dev/mapper/c338434f254439c7cd0c8ba21a872405 to /run/swift-storage/c338434f254439c7cd0c8ba21a872405\r\n2016/11/28 14:11:59 INFO: mounted /dev/mapper/81dde30aa3d9bf946fb0281f260b68e4 to /srv/node/swift-01\r\n2016/11/28 14:11:59 DEBUG: chown /srv/node/swift-01 to 1000:1000\r\n2016/11/28 14:11:59 INFO: unmounted /run/swift-storage/81dde30aa3d9bf946fb0281f260b68e4\r\n2016/11/28 14:11:59 INFO: mounted /dev/mapper/dbc0b33565fc9c0784c8ed66ec7d123e to /srv/node/swift-03\r\n2016/11/28 14:11:59 DEBUG: chown /srv/node/swift-03 to 1000:1000\r\n2016/11/28 14:11:59 INFO: unmounted /run/swift-storage/dbc0b33565fc9c0784c8ed66ec7d123e\r\n2016/11/28 14:11:59 INFO: mounted /dev/mapper/c338434f254439c7cd0c8ba21a872405 to /srv/node/swift-02\r\n2016/11/28 14:11:59 DEBUG: chown /srv/node/swift-02 to 1000:1000\r\n2016/11/28 14:11:59 INFO: unmounted /run/swift-storage/c338434f254439c7cd0c8ba21a872405\r\n2016/11/28 14:12:28 DEBUG: ActiveLUKSMappings = map[string]string{\"/dev/sdc\":\"81dde30aa3d9bf946fb0281f260b68e4\", \"/dev/sde\":\"c338434f254439c7cd0c8ba21a872405\", \"/dev/sdd\":\"dbc0b33565fc9c0784c8ed66ec7d123e\"}\r\n2016/11/28 14:12:28 INFO: event received: scheduled consistency check\r\n2016/11/28 14:12:28 ERROR: expected /dev/mapper/81dde30aa3d9bf946fb0281f260b68e4 to be mounted at /srv/node/swift-01, but is not mounted anymore\r\n2016/11/28 14:12:28 INFO: flagging /dev/sdc as broken because of previous error\r\n2016/11/28 14:12:28 INFO: To reinstate this drive into the cluster, delete the symlink at /run/swift-storage/broken/81dde30aa3d9bf946fb0281f260b68e4\r\n2016/11/28 14:12:28 ERROR: expected /dev/mapper/dbc0b33565fc9c0784c8ed66ec7d123e to be mounted at /srv/node/swift-03, but is not mounted anymore\r\n2016/11/28 14:12:28 INFO: flagging /dev/sdd as broken because of previous error\r\n2016/11/28 14:12:28 INFO: To reinstate this drive into the cluster, delete the symlink at /run/swift-storage/broken/dbc0b33565fc9c0784c8ed66ec7d123e\r\n2016/11/28 14:12:28 ERROR: expected /dev/mapper/c338434f254439c7cd0c8ba21a872405 to be mounted at /srv/node/swift-02, but is not mounted anymore\r\n2016/11/28 14:12:28 INFO: flagging /dev/sde as broken because of previous error\r\n2016/11/28 14:12:28 INFO: To reinstate this drive into the cluster, delete the symlink at /run/swift-storage/broken/c338434f254439c7cd0c8ba21a872405\r\n```\r\n\r\nThis is interesting. I would have expected `ActiveMounts = []` in the second convergence, but not in the first one."}}
{"actor":{"display_login":"majewsky"}}