{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/70730","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/70730/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/70730/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/70730/events","html_url":"https://github.com/kubernetes/kubernetes/issues/70730","id":378165283,"node_id":"MDU6SXNzdWUzNzgxNjUyODM=","number":70730,"title":"kube-scheduler die because of lost lease while kube-apiserver is still working","user":{"login":"gbjuno","id":2673832,"node_id":"MDQ6VXNlcjI2NzM4MzI=","avatar_url":"https://avatars2.githubusercontent.com/u/2673832?v=4","gravatar_id":"","url":"https://api.github.com/users/gbjuno","html_url":"https://github.com/gbjuno","followers_url":"https://api.github.com/users/gbjuno/followers","following_url":"https://api.github.com/users/gbjuno/following{/other_user}","gists_url":"https://api.github.com/users/gbjuno/gists{/gist_id}","starred_url":"https://api.github.com/users/gbjuno/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gbjuno/subscriptions","organizations_url":"https://api.github.com/users/gbjuno/orgs","repos_url":"https://api.github.com/users/gbjuno/repos","events_url":"https://api.github.com/users/gbjuno/events{/privacy}","received_events_url":"https://api.github.com/users/gbjuno/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":125550211,"node_id":"MDU6TGFiZWwxMjU1NTAyMTE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/scheduling","name":"sig/scheduling","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-11-07T07:01:47Z","updated_at":"2018-12-27T12:57:44Z","closed_at":null,"author_association":"NONE","body":"<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!-->\r\n**What keywords did you search in Kubernetes issues before filing this one?** (If you have found any duplicates, you should instead reply there.):\r\nkube-scheduler, scheduler, dies, lost, master\r\n\r\n**What happened**:\r\nThe kube-scheduler process dies in a cluster which lost two of three master nodes while kube-apiserver is still working. All the three master nodes were running etcd/kube-apiserver/controller/scheduler components. \r\n\r\n```\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.340805   30766 leaderelection.go:243] lock is held by 10-2-146-46_e3be14f7-e256-11e8-ae98-0050569e4dac and has not yet expired\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.340874   30766 round_trippers.go:384] GET http://10.2.146.250:8080/api/v1/namespaces/kube-system/endpoints/kube-scheduler\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.340882   30766 round_trippers.go:391] Request Headers:\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.340886   30766 round_trippers.go:394]     Accept: application/vnd.kubernetes.protobuf, */*\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.340891   30766 round_trippers.go:394]     User-Agent: kube-scheduler/v1.10.8 (linux/amd64) kubernetes/7eab6a4/leader-election\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.342930   30766 round_trippers.go:409] Response Status: 200 OK in 2 milliseconds\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.342949   30766 round_trippers.go:412] Response Headers:\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.342955   30766 round_trippers.go:415]     Content-Type: application/vnd.kubernetes.protobuf\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.342959   30766 round_trippers.go:415]     Date: Wed, 07 Nov 2018 06:37:56 GMT\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.342964   30766 round_trippers.go:415]     Content-Length: 415\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.343034   30766 request.go:872] Response Body:\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000000  6b 38 73 00 0a 0f 0a 02  76 31 12 09 45 6e 64 70  |k8s.....v1..Endp|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000010  6f 69 6e 74 73 12 83 03  0a 80 03 0a 0e 6b 75 62  |oints........kub|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000020  65 2d 73 63 68 65 64 75  6c 65 72 12 00 1a 0b 6b  |e-scheduler....k|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000030  75 62 65 2d 73 79 73 74  65 6d 22 37 2f 61 70 69  |ube-system\"7/api|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000040  2f 76 31 2f 6e 61 6d 65  73 70 61 63 65 73 2f 6b  |/v1/namespaces/k|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000050  75 62 65 2d 73 79 73 74  65 6d 2f 65 6e 64 70 6f  |ube-system/endpo|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000060  69 6e 74 73 2f 6b 75 62  65 2d 73 63 68 65 64 75  |ints/kube-schedu|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000070  6c 65 72 2a 24 61 64 31  65 61 39 64 32 2d 65 32  |ler*$ad1ea9d2-e2|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000080  33 64 2d 31 31 65 38 2d  38 66 30 38 2d 30 30 35  |3d-11e8-8f08-005|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000090  30 35 36 39 65 66 34 61  33 32 05 31 36 39 38 36  |0569ef4a32.16986|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 000000a0  38 00 42 08 08 ac b9 89  df 05 10 00 62 ea 01 0a  |8.B.........b...|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 000000b0  28 63 6f 6e 74 72 6f 6c  2d 70 6c 61 6e 65 2e 61  |(control-plane.a|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 000000c0  6c 70 68 61 2e 6b 75 62  65 72 6e 65 74 65 73 2e  |lpha.kubernetes [truncated 1029 chars]\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.343119   30766 leaderelection.go:243] lock is held by 10-2-146-46_e3be14f7-e256-11e8-ae98-0050569e4dac and has not yet expired\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.343175   30766 leaderelection.go:203] failed to renew lease kube-system/kube-scheduler: timed out waiting for the condition\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: E1107 14:37:56.343201   30766 server.go:612] lost master\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: F1107 14:37:56.343230   30766 helpers.go:119] error: lost lease\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: I1107 14:37:56.344868   30766 request.go:872] Request Body:\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000000  6b 38 73 00 0a 0b 0a 02  76 31 12 05 45 76 65 6e  |k8s.....v1..Even|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000010  74 12 aa 02 0a 3c 0a 1f  6b 75 62 65 2d 73 63 68  |t....<..kube-sch|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000020  65 64 75 6c 65 72 2e 31  35 36 34 63 34 32 65 37  |eduler.1564c42e7|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000030  38 65 31 62 37 64 33 12  00 1a 0b 6b 75 62 65 2d  |8e1b7d3....kube-|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000040  73 79 73 74 65 6d 22 00  2a 00 32 00 38 00 42 00  |system\".*.2.8.B.|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000050  7a 00 12 5b 0a 09 45 6e  64 70 6f 69 6e 74 73 12  |z..[..Endpoints.|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000060  0b 6b 75 62 65 2d 73 79  73 74 65 6d 1a 0e 6b 75  |.kube-system..ku|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000070  62 65 2d 73 63 68 65 64  75 6c 65 72 22 24 61 64  |be-scheduler\"$ad|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000080  31 65 61 39 64 32 2d 65  32 33 64 2d 31 31 65 38  |1ea9d2-e23d-11e8|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 00000090  2d 38 66 30 38 2d 30 30  35 30 35 36 39 65 66 34  |-8f08-0050569ef4|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 000000a0  61 33 2a 02 76 31 32 05  31 36 39 38 36 3a 00 1a  |a3*.v12.16986:..|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 000000b0  0e 4c 65 61 64 65 72 45  6c 65 63 74 69 6f 6e 22  |.LeaderElection\"|\r\nNov  7 14:37:56 10-2-146-48 kube-scheduler[30766]: 000000c0  40 31 30 2d 32 2d 31 34  36 2d 34 38 5f 37 33 32  |@10-2-146-48_73 [truncated 621 chars]\r\n```\r\n\r\n**What you expected to happen**:\r\nkube-scheduler should renew the leader and continue running \r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\nkill two of three master nodes, kube-scheduler running in the healthy one dies.\r\n\r\n**Anything else we need to know?**:\r\nAt the same time , kube-apiserver is still working.\r\n```\r\nroot@10-2-146-48:/var/log/upstart# curl http://10.2.146.250:8080/api/v1/namespaces/kube-system/endpoints/kube-scheduler\r\n{\r\n  \"kind\": \"Endpoints\",\r\n  \"apiVersion\": \"v1\",\r\n  \"metadata\": {\r\n    \"name\": \"kube-scheduler\",\r\n    \"namespace\": \"kube-system\",\r\n    \"selfLink\": \"/api/v1/namespaces/kube-system/endpoints/kube-scheduler\",\r\n    \"uid\": \"ad1ea9d2-e23d-11e8-8f08-0050569ef4a3\",\r\n    \"resourceVersion\": \"16986\",\r\n    \"creationTimestamp\": \"2018-11-07T03:31:56Z\",\r\n    \"annotations\": {\r\n      \"control-plane.alpha.kubernetes.io/leader\": \"{\\\"holderIdentity\\\":\\\"10-2-146-46_e3be14f7-e256-11e8-ae98-0050569e4dac\\\",\\\"leaseDurationSeconds\\\":15,\\\"acquireTime\\\":\\\"2018-11-07T06:32:47Z\\\",\\\"renewTime\\\":\\\"2018-11-07T06:36:04Z\\\",\\\"leaderTransitions\\\":6}\"\r\n    }\r\n  }\r\n```\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`): v1.10.8\r\n- Cloud provider or hardware configuration:  ubuntu\r\n- OS (e.g. from /etc/os-release): ubuntu 16.04\r\n- Kernel (e.g. `uname -a`): Linux 10-2-146-48 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n- Install tools: binary run\r\n- Others:\r\netcd cluster (5 nodes); \r\nkube-apiserver/controller/scheduler(3 nodes + vip + haproxy in master-backup mode serving kube-apiserver)\r\n\r\n<!-- DO NOT EDIT BELOW THIS LINE -->\r\n/kind bug"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/450146100","html_url":"https://github.com/kubernetes/kubernetes/issues/70730#issuecomment-450146100","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/70730","id":450146100,"node_id":"MDEyOklzc3VlQ29tbWVudDQ1MDE0NjEwMA==","user":{"login":"uablrek","id":37046727,"node_id":"MDQ6VXNlcjM3MDQ2NzI3","avatar_url":"https://avatars1.githubusercontent.com/u/37046727?v=4","gravatar_id":"","url":"https://api.github.com/users/uablrek","html_url":"https://github.com/uablrek","followers_url":"https://api.github.com/users/uablrek/followers","following_url":"https://api.github.com/users/uablrek/following{/other_user}","gists_url":"https://api.github.com/users/uablrek/gists{/gist_id}","starred_url":"https://api.github.com/users/uablrek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uablrek/subscriptions","organizations_url":"https://api.github.com/users/uablrek/orgs","repos_url":"https://api.github.com/users/uablrek/repos","events_url":"https://api.github.com/users/uablrek/events{/privacy}","received_events_url":"https://api.github.com/users/uablrek/received_events","type":"User","site_admin":false},"created_at":"2018-12-27T12:57:44Z","updated_at":"2018-12-27T12:57:44Z","author_association":"CONTRIBUTOR","body":"I have seen this also but while experimenting with etcd `v3.3.10`."}}
{"actor":{"display_login":"uablrek"}}