{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/acs-engine/issues/2892","repository_url":"https://api.github.com/repos/Azure/acs-engine","labels_url":"https://api.github.com/repos/Azure/acs-engine/issues/2892/labels{/name}","comments_url":"https://api.github.com/repos/Azure/acs-engine/issues/2892/comments","events_url":"https://api.github.com/repos/Azure/acs-engine/issues/2892/events","html_url":"https://github.com/Azure/acs-engine/issues/2892","id":321647279,"number":2892,"title":"ReadinessProbe / LivenessProbe critically failing in the entire cluster","user":{"login":"hmarcelodn","id":5897525,"avatar_url":"https://avatars3.githubusercontent.com/u/5897525?v=4","gravatar_id":"","url":"https://api.github.com/users/hmarcelodn","html_url":"https://github.com/hmarcelodn","followers_url":"https://api.github.com/users/hmarcelodn/followers","following_url":"https://api.github.com/users/hmarcelodn/following{/other_user}","gists_url":"https://api.github.com/users/hmarcelodn/gists{/gist_id}","starred_url":"https://api.github.com/users/hmarcelodn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hmarcelodn/subscriptions","organizations_url":"https://api.github.com/users/hmarcelodn/orgs","repos_url":"https://api.github.com/users/hmarcelodn/repos","events_url":"https://api.github.com/users/hmarcelodn/events{/privacy}","received_events_url":"https://api.github.com/users/hmarcelodn/received_events","type":"User","site_admin":false},"labels":[{"id":460646258,"url":"https://api.github.com/repos/Azure/acs-engine/labels/orchestrator/k8s","name":"orchestrator/k8s","color":"2e64d2","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-09T16:51:50Z","updated_at":"2018-05-11T00:15:21Z","closed_at":null,"author_association":"NONE","body":"<!-- Thanks for filing an issue! Before hitting the button, please answer these questions. It's helpful to search the existing GitHub issues first. It's likely that another user has already reported the issue you're facing, or it's a known issue that we're already aware of-->\r\n\r\n**Is this a request for help?**:\r\n\r\nYes\r\n\r\n---\r\n\r\n**Is this an ISSUE or FEATURE REQUEST?** (choose one):\r\n\r\nISSUE\r\n\r\n---\r\n\r\n**What version of acs-engine?**:\r\n\r\n0.13.1\r\n\r\n---\r\n\r\n\r\n<!--\r\nIf this is a ISSUE, please:\r\n  - Fill in as much of the template below as you can.  If you leave out\r\n    information, we can't help you as well.\r\n\r\nIf this is a FEATURE REQUEST, please:\r\n  - Describe *in detail* the feature/behavior/change you'd like to see.\r\n\r\nIn both cases, be ready for followup questions, and please respond in a timely\r\nmanner.  If we can't reproduce a bug or think a feature already exists, we\r\nmight close your issue.  If we're wrong, PLEASE feel free to reopen it and\r\nexplain why.\r\n-->\r\n\r\n**Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)**\r\n\r\n1.6.6\r\n\r\n**What happened**:\r\n\r\nI have 6 clusters created the past year, recently we updated (in January) the version of acs-engine from 0.5.0 to 0.13.1 and the cluster was working properly. Suddenly all my 6 clusters are down because Readiness/Liveness probes are failing everywhere, from kube-system pods to the entire pods causing Termination on all the rest of pods. This is very very critical issue. It seems to be solved after restarting nodes, but after a few minutes the chaos starts again.\r\n\r\n**What you expected to happen**:\r\n\r\nKeep working.\r\n\r\n**How to reproduce it** (as minimally and precisely as possible):\r\n\r\nDeploy a 1.6.x latest version with acs-engine 0.13.1.\r\n\r\n**Anything else we need to know**:\r\n\r\nI have seen many issues related to this. My pods now:\r\n\r\n\r\n> tiller-deploy-3575610951-ggk0x k8s-agentpool1-15972647-0\r\n> Readiness probe failed: Get http://10.244.1.92:44135/readiness: net/http: request canceled (Client.Timeout exceeded while awaiting headers)\r\n> Error syncing pod, skipping: rpc error: code = 4 desc = context deadline exceeded\r\n\r\n \r\n\r\n> kube-dns-v20-2135026167-9hds5 k8s-agentpool1-15972647-0\r\n> Liveness probe failed: Get http://10.244.1.91:8080/healthz-dnsmasq: net/http: request canceled (Client.Timeout exceeded while awaiting headers)\r\n> Failed to create container with error: rpc error: code = 4 desc = context deadline exceeded\r\n> Error syncing pod, skipping: failed to \"StartContainer\" for \"healthz\" with rpc error: code = 4 desc = context deadline exceeded: \"Create Container Failed\" "},"comment":{"url":"https://api.github.com/repos/Azure/acs-engine/issues/comments/388222847","html_url":"https://github.com/Azure/acs-engine/issues/2892#issuecomment-388222847","issue_url":"https://api.github.com/repos/Azure/acs-engine/issues/2892","id":388222847,"user":{"login":"jackfrancis","id":1494193,"avatar_url":"https://avatars0.githubusercontent.com/u/1494193?v=4","gravatar_id":"","url":"https://api.github.com/users/jackfrancis","html_url":"https://github.com/jackfrancis","followers_url":"https://api.github.com/users/jackfrancis/followers","following_url":"https://api.github.com/users/jackfrancis/following{/other_user}","gists_url":"https://api.github.com/users/jackfrancis/gists{/gist_id}","starred_url":"https://api.github.com/users/jackfrancis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jackfrancis/subscriptions","organizations_url":"https://api.github.com/users/jackfrancis/orgs","repos_url":"https://api.github.com/users/jackfrancis/repos","events_url":"https://api.github.com/users/jackfrancis/events{/privacy}","received_events_url":"https://api.github.com/users/jackfrancis/received_events","type":"User","site_admin":false},"created_at":"2018-05-11T00:15:21Z","updated_at":"2018-05-11T00:15:21Z","author_association":"MEMBER","body":"FYI I have a PR that I'm using to add environmental DNS sanity checks for our E2E cluster deployments:\r\n\r\nhttps://github.com/Azure/acs-engine/pull/2921\r\n\r\nPlease add any high level wishlist items or concrete coverage requests there to get ahead of these issues going forward. Thanks!"}}
{"actor":{"display_login":"jackfrancis"}}