{"action":"created","issue":{"url":"https://api.github.com/repos/cybozu-go/topolvm/issues/57","repository_url":"https://api.github.com/repos/cybozu-go/topolvm","labels_url":"https://api.github.com/repos/cybozu-go/topolvm/issues/57/labels{/name}","comments_url":"https://api.github.com/repos/cybozu-go/topolvm/issues/57/comments","events_url":"https://api.github.com/repos/cybozu-go/topolvm/issues/57/events","html_url":"https://github.com/cybozu-go/topolvm/issues/57","id":500521403,"node_id":"MDU6SXNzdWU1MDA1MjE0MDM=","number":57,"title":"Default webhook configuration can result in cluster wide deadlock","user":{"login":"matthias50","id":24896067,"node_id":"MDQ6VXNlcjI0ODk2MDY3","avatar_url":"https://avatars0.githubusercontent.com/u/24896067?v=4","gravatar_id":"","url":"https://api.github.com/users/matthias50","html_url":"https://github.com/matthias50","followers_url":"https://api.github.com/users/matthias50/followers","following_url":"https://api.github.com/users/matthias50/following{/other_user}","gists_url":"https://api.github.com/users/matthias50/gists{/gist_id}","starred_url":"https://api.github.com/users/matthias50/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthias50/subscriptions","organizations_url":"https://api.github.com/users/matthias50/orgs","repos_url":"https://api.github.com/users/matthias50/repos","events_url":"https://api.github.com/users/matthias50/events{/privacy}","received_events_url":"https://api.github.com/users/matthias50/received_events","type":"User","site_admin":false},"labels":[{"id":1336422897,"node_id":"MDU6TGFiZWwxMzM2NDIyODk3","url":"https://api.github.com/repos/cybozu-go/topolvm/labels/bug","name":"bug","color":"d73a4a","default":true}],"state":"open","locked":false,"assignee":{"login":"ymmt2005","id":1461515,"node_id":"MDQ6VXNlcjE0NjE1MTU=","avatar_url":"https://avatars3.githubusercontent.com/u/1461515?v=4","gravatar_id":"","url":"https://api.github.com/users/ymmt2005","html_url":"https://github.com/ymmt2005","followers_url":"https://api.github.com/users/ymmt2005/followers","following_url":"https://api.github.com/users/ymmt2005/following{/other_user}","gists_url":"https://api.github.com/users/ymmt2005/gists{/gist_id}","starred_url":"https://api.github.com/users/ymmt2005/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ymmt2005/subscriptions","organizations_url":"https://api.github.com/users/ymmt2005/orgs","repos_url":"https://api.github.com/users/ymmt2005/repos","events_url":"https://api.github.com/users/ymmt2005/events{/privacy}","received_events_url":"https://api.github.com/users/ymmt2005/received_events","type":"User","site_admin":false},"assignees":[{"login":"ymmt2005","id":1461515,"node_id":"MDQ6VXNlcjE0NjE1MTU=","avatar_url":"https://avatars3.githubusercontent.com/u/1461515?v=4","gravatar_id":"","url":"https://api.github.com/users/ymmt2005","html_url":"https://github.com/ymmt2005","followers_url":"https://api.github.com/users/ymmt2005/followers","following_url":"https://api.github.com/users/ymmt2005/following{/other_user}","gists_url":"https://api.github.com/users/ymmt2005/gists{/gist_id}","starred_url":"https://api.github.com/users/ymmt2005/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ymmt2005/subscriptions","organizations_url":"https://api.github.com/users/ymmt2005/orgs","repos_url":"https://api.github.com/users/ymmt2005/repos","events_url":"https://api.github.com/users/ymmt2005/events{/privacy}","received_events_url":"https://api.github.com/users/ymmt2005/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2019-09-30T20:58:37Z","updated_at":"2019-10-02T20:57:19Z","closed_at":null,"author_association":"NONE","body":"**Describe the bug**\r\nThe default setup for the Deployment of topolvm-hook has the webhook running on nodes and only ignoring pods in topolvm-system. This can lead to cluster wide deadlocks if an overlay network such as Calico is in use if all nodes go down at the same time. \r\n\r\n**Environments**\r\n- Version: Topolvm 0.1.2, Kubernetes 1.14.7, Calico 3.7.3\r\n- OS: N/A\r\n\r\n**To Reproduce**\r\nSteps to reproduce the behavior:\r\n1. Deploy topolvm\r\n2. Restart all the worker nodes (non master nodes) in the cluster\r\n\r\n**Expected behavior**\r\nAll nodes come back to the \"Ready\" state.\r\n\r\n**Additional context**\r\nThe nodes are all stuck in NotReady because none of the Calico daemonset pods can be scheduled because the topolvm-hook webhook can't execute and the topolvm-hook pods can't be scheduled because all nodes are in NotReady because Calico isn't running on the nodes.\r\n\r\nI think it may be best by default to modify the namespaceSelector in the MutatingWebhookConfiguration for topolvm-hook to exclude both topolvm-system and kube-system. It might also be worth adding a warning in the documentation to exclude any namespace where an overlay network is running in case someone is using a namespace other than kube-system for the same purpose.\r\n"},"comment":{"url":"https://api.github.com/repos/cybozu-go/topolvm/issues/comments/537677403","html_url":"https://github.com/cybozu-go/topolvm/issues/57#issuecomment-537677403","issue_url":"https://api.github.com/repos/cybozu-go/topolvm/issues/57","id":537677403,"node_id":"MDEyOklzc3VlQ29tbWVudDUzNzY3NzQwMw==","user":{"login":"matthias50","id":24896067,"node_id":"MDQ6VXNlcjI0ODk2MDY3","avatar_url":"https://avatars0.githubusercontent.com/u/24896067?v=4","gravatar_id":"","url":"https://api.github.com/users/matthias50","html_url":"https://github.com/matthias50","followers_url":"https://api.github.com/users/matthias50/followers","following_url":"https://api.github.com/users/matthias50/following{/other_user}","gists_url":"https://api.github.com/users/matthias50/gists{/gist_id}","starred_url":"https://api.github.com/users/matthias50/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthias50/subscriptions","organizations_url":"https://api.github.com/users/matthias50/orgs","repos_url":"https://api.github.com/users/matthias50/repos","events_url":"https://api.github.com/users/matthias50/events{/privacy}","received_events_url":"https://api.github.com/users/matthias50/received_events","type":"User","site_admin":false},"created_at":"2019-10-02T20:57:19Z","updated_at":"2019-10-02T20:57:19Z","author_association":"NONE","body":"So I did something slightly different. I *terminated* in all the worker nodes in the cluster and the AWS ASGs recreated them. In this case, the pods are not yet defined and the daemonset controller can't create them."}}
{"actor":{"display_login":"matthias50"}}