{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/69786","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/69786/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/69786/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/69786/events","html_url":"https://github.com/kubernetes/kubernetes/issues/69786","id":369928605,"node_id":"MDU6SXNzdWUzNjk5Mjg2MDU=","number":69786,"title":"[sig-cluster-lifecycle] Reboot [Disruptive] [Feature:Reboot] each node by ordering unclean reboot and ensure they function upon restartChanges","user":{"login":"mortent","id":8463,"node_id":"MDQ6VXNlcjg0NjM=","avatar_url":"https://avatars2.githubusercontent.com/u/8463?v=4","gravatar_id":"","url":"https://api.github.com/users/mortent","html_url":"https://github.com/mortent","followers_url":"https://api.github.com/users/mortent/followers","following_url":"https://api.github.com/users/mortent/following{/other_user}","gists_url":"https://api.github.com/users/mortent/gists{/gist_id}","starred_url":"https://api.github.com/users/mortent/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mortent/subscriptions","organizations_url":"https://api.github.com/users/mortent/orgs","repos_url":"https://api.github.com/users/mortent/repos","events_url":"https://api.github.com/users/mortent/events{/privacy}","received_events_url":"https://api.github.com/users/mortent/received_events","type":"User","site_admin":false},"labels":[{"id":496752468,"node_id":"MDU6TGFiZWw0OTY3NTI0Njg=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/failing-test","name":"kind/failing-test","color":"e11d21","default":false},{"id":114528223,"node_id":"MDU6TGFiZWwxMTQ1MjgyMjM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/important-soon","name":"priority/important-soon","color":"eb6420","default":false},{"id":173494222,"node_id":"MDU6TGFiZWwxNzM0OTQyMjI=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/cluster-lifecycle","name":"sig/cluster-lifecycle","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-10-14T18:16:51Z","updated_at":"2018-10-15T22:02:22Z","closed_at":null,"author_association":"MEMBER","body":"Reboot tests have been failing since 10/12\r\n\r\nDashboard: https://k8s-testgrid.appspot.com/sig-release-master-blocking#gke-cos-master-reboot\r\n\r\nSample Failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/18153\r\n\r\nThis test failure will block 1.13 beta, so please let's fix it before then. Thanks!\r\n\r\n/sig cluster-lifecycle\r\n/kind failing-test\r\n/priority important-soon"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/430030099","html_url":"https://github.com/kubernetes/kubernetes/issues/69786#issuecomment-430030099","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/69786","id":430030099,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMDAzMDA5OQ==","user":{"login":"msau42","id":24448061,"node_id":"MDQ6VXNlcjI0NDQ4MDYx","avatar_url":"https://avatars3.githubusercontent.com/u/24448061?v=4","gravatar_id":"","url":"https://api.github.com/users/msau42","html_url":"https://github.com/msau42","followers_url":"https://api.github.com/users/msau42/followers","following_url":"https://api.github.com/users/msau42/following{/other_user}","gists_url":"https://api.github.com/users/msau42/gists{/gist_id}","starred_url":"https://api.github.com/users/msau42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msau42/subscriptions","organizations_url":"https://api.github.com/users/msau42/orgs","repos_url":"https://api.github.com/users/msau42/repos","events_url":"https://api.github.com/users/msau42/events{/privacy}","received_events_url":"https://api.github.com/users/msau42/received_events","type":"User","site_admin":false},"created_at":"2018-10-15T22:02:22Z","updated_at":"2018-10-15T22:02:22Z","author_association":"MEMBER","body":"Copying my notes here:\r\n\r\nLooking at one recent test run:\r\nhttps://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/18231/build-log.txt\r\n\r\nThe test failed because kubelet never became \"not ready\":\r\n```\r\nI1015 19:33:01.392] Oct 15 19:33:01.390: INFO: Waiting up to 2m0s for node gke-e2e-18231-344e3-default-pool-2b8d22c7-njwh condition Ready to be false\r\nI1015 19:33:01.393] Oct 15 19:33:01.393: INFO: Condition Ready of node gke-e2e-18231-344e3-default-pool-2b8d22c7-njwh is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled\r\n...\r\nI1015 19:34:59.825] Oct 15 19:34:59.825: INFO: Condition Ready of node gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled\r\nI1015 19:34:59.826] Oct 15 19:34:59.825: INFO: Condition Ready of node gke-e2e-18231-344e3-default-pool-2b8d22c7-q5g8 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled\r\nI1015 19:34:59.826] Oct 15 19:34:59.825: INFO: Condition Ready of node gke-e2e-18231-344e3-default-pool-2b8d22c7-njwh is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status. AppArmor enabled\r\nI1015 19:35:01.825] Oct 15 19:35:01.825: INFO: Node gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr didn't reach desired Ready condition status (false) within 2m0s\r\nI1015 19:35:01.826] Oct 15 19:35:01.825: INFO: Node gke-e2e-18231-344e3-default-pool-2b8d22c7-q5g8 didn't reach desired Ready condition status (false) within 2m0s\r\nI1015 19:35:01.830] Oct 15 19:35:01.826: INFO: Node gke-e2e-18231-344e3-default-pool-2b8d22c7-njwh didn't reach desired Ready condition status (false) within 2m0s\r\nI1015 19:35:01.830] Oct 15 19:35:01.826: INFO: Node gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr failed reboot test.\r\nI1015 19:35:01.830] Oct 15 19:35:01.826: INFO: Node gke-e2e-18231-344e3-default-pool-2b8d22c7-njwh failed reboot test.\r\nI1015 19:35:01.830] Oct 15 19:35:01.826: INFO: Node gke-e2e-18231-344e3-default-pool-2b8d22c7-q5g8 failed reboot test.\r\n```\r\n\r\nWhen looking at one of the kubelet logs, I do see that kubelet had problems making Node updates to the api server:\r\nhttps://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/18231/artifacts/gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr/kubelet.log\r\n```\r\nE1015 19:33:26.150380    1303 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr\": Get https://35.202.63.115/api/v1/nodes/gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr?resourceVersion=0&timeout=10s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)\r\nI1015 19:33:29.552319    1303 cloud_request_manager.go:113] Node addresses from cloud provider for node \"gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr\" not collected\r\nE1015 19:33:36.150912    1303 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr\": Get https://35.202.63.115/api/v1/nodes/gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr?timeout=10s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)\r\nI1015 19:33:41.553161    1303 cloud_request_manager.go:113] Node addresses from cloud provider for node \"gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr\" not collected\r\nE1015 19:33:46.152035    1303 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr\": Get https://35.202.63.115/api/v1/nodes/gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr?timeout=10s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\r\nI1015 19:33:53.554253    1303 cloud_request_manager.go:113] Node addresses from cloud provider for node \"gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr\" not collected\r\nE1015 19:33:56.152555    1303 kubelet_node_status.go:377] Error updating node status, will retry: error getting node \"gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr\": Get https://35.202.63.115/api/v1/nodes/gke-e2e-18231-344e3-default-pool-2b8d22c7-01rr?timeout=10s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\r\n```\r\n\r\nErrors continue until kubelet is restarted 8 minutes later (the test times out at 2 minutes):\r\n```\r\nStopping Kubernetes kubelet...\r\nStopped Kubernetes kubelet.\r\nStarted Kubernetes kubelet.\r\nFlag --allow-privileged has been deprecated, will be removed in a future version\r\nFlag --max-pods has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r\nI1015 19:41:30.505574    1169 flags.go:33] FLAG: --address=\"0.0.0.0\"\r\n```\r\n"}}
{"actor":{"display_login":"msau42"}}