{"action":"opened","issue":{"url":"https://api.github.com/repos/heketi/heketi/issues/1403","repository_url":"https://api.github.com/repos/heketi/heketi","labels_url":"https://api.github.com/repos/heketi/heketi/issues/1403/labels{/name}","comments_url":"https://api.github.com/repos/heketi/heketi/issues/1403/comments","events_url":"https://api.github.com/repos/heketi/heketi/issues/1403/events","html_url":"https://github.com/heketi/heketi/issues/1403","id":373100502,"node_id":"MDU6SXNzdWUzNzMxMDA1MDI=","number":1403,"title":"heketidbstorage not mounted on reboot, empty dir used instead and blank db is created.","user":{"login":"howels","id":7933842,"node_id":"MDQ6VXNlcjc5MzM4NDI=","avatar_url":"https://avatars1.githubusercontent.com/u/7933842?v=4","gravatar_id":"","url":"https://api.github.com/users/howels","html_url":"https://github.com/howels","followers_url":"https://api.github.com/users/howels/followers","following_url":"https://api.github.com/users/howels/following{/other_user}","gists_url":"https://api.github.com/users/howels/gists{/gist_id}","starred_url":"https://api.github.com/users/howels/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/howels/subscriptions","organizations_url":"https://api.github.com/users/howels/orgs","repos_url":"https://api.github.com/users/howels/repos","events_url":"https://api.github.com/users/howels/events{/privacy}","received_events_url":"https://api.github.com/users/howels/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-23T16:56:05Z","updated_at":"2018-10-23T16:56:05Z","closed_at":null,"author_association":"NONE","body":"Heketi mounts empty directory instead of the DB PV, then creates a blank DB. This appears to occur when a node is rebooted. The DB still exists on the PV, but as Heketi is no longer seeing the DB then future attempts to create, mount or delete volumes all fail. This leads to a condition where restarting the Heketi pod will also fail to mount the volume and the service is deadlocked.\r\n\r\n### Kind of issue\r\n\r\nBug\r\n\r\n### Observed behavior\r\n\r\nHeketiDbStorage is mounted as an empty dir and a blank DB is created instead of the actual DB being used.\r\n\r\n### Expected/desired behavior\r\n\r\nIf PV mount fails then the Pod should not be scheduled. The Heketi Pod should only start when heketidbstorage is available and mounted.\r\n\r\n### Details on how to reproduce (minimal and precise)\r\n\r\nUnfortunately this appears to occur randomly on some node reboots. The issue is hard to reproduce but appears to be a timing issue. Nonetheless Pods should not be scheduled with unavailable mounts.\r\n\r\n### Information about the environment:\r\n\r\n- Heketi version used (e.g. v6.0.0 or master): \r\n4.1\r\n- Operating system used: \r\nCentOS 7.5\r\n- Heketi compiled from sources, as a package (rpm/deb), or container: \r\nDeployed with latest gk-deploy\r\n- If container, which container image: \r\nLatest gk-deploy\r\n- Using kubernetes, openshift, or direct install: \r\nKubbernetes\r\n- If kubernetes/openshift, is gluster running inside kubernetes/openshift or outside: \r\nInside\r\n- If kubernetes/openshift, how was it deployed (gk-deploy, openshift-ansible, other, custom): \r\ngk-deploy\r\n\r\n\r\n### Other useful information\r\n\r\n\r\n"}}
{"actor":{"display_login":"howels"}}