{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/9668","repository_url":"https://api.github.com/repos/coreos/etcd","labels_url":"https://api.github.com/repos/coreos/etcd/issues/9668/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/9668/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/9668/events","html_url":"https://github.com/coreos/etcd/issues/9668","id":319087200,"number":9668,"title":"etcd cluster no leader failure","user":{"login":"JuneZhao","id":1699200,"avatar_url":"https://avatars1.githubusercontent.com/u/1699200?v=4","gravatar_id":"","url":"https://api.github.com/users/JuneZhao","html_url":"https://github.com/JuneZhao","followers_url":"https://api.github.com/users/JuneZhao/followers","following_url":"https://api.github.com/users/JuneZhao/following{/other_user}","gists_url":"https://api.github.com/users/JuneZhao/gists{/gist_id}","starred_url":"https://api.github.com/users/JuneZhao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuneZhao/subscriptions","organizations_url":"https://api.github.com/users/JuneZhao/orgs","repos_url":"https://api.github.com/users/JuneZhao/repos","events_url":"https://api.github.com/users/JuneZhao/events{/privacy}","received_events_url":"https://api.github.com/users/JuneZhao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-01T01:13:17Z","updated_at":"2018-05-02T20:41:44Z","closed_at":null,"author_association":"NONE","body":"We repeatedly receive below error on our k8s cluster on etcd \r\n\r\nE0429 15:54:53.235735       1 status.go:62] apiserver received an error that is not an metav1.Status: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 19:35:17.626027       1 status.go:62] apiserver received an error that is not an metav1.Status: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\n \r\nThe controller-manager and scheduler repeatedly report this error:\r\ncommand kubectl logs kube-controller-manager-k8s-master-3688e46e-0 -n kube-system\r\n2018-04-29 15:54:55.424209 I | proto: duplicate proto type registered: google.protobuf.Any\r\n2018-04-29 15:54:55.424350 I | proto: duplicate proto type registered: google.protobuf.Duration\r\n2018-04-29 15:54:55.424383 I | proto: duplicate proto type registered: google.protobuf.Timestamp\r\nI0429 15:54:55.478452       1 controllermanager.go:108] Version: v1.7.7\r\nI0429 15:54:55.482362       1 leaderelection.go:179] attempting to acquire leader lease...\r\nE0429 16:00:54.906678       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 16:21:09.870318       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 16:54:15.669323       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 17:27:01.326286       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 17:45:59.433418       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 19:35:18.510143       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 19:39:06.037933       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 19:44:28.138011       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 19:44:41.994036       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 19:51:28.825744       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 21:04:53.971803       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 21:07:39.440264       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 22:07:37.679411       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0429 22:09:39.748550       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\n \r\n \r\ncommand kubectl logs kube-scheduler-k8s-master-3688e46e-0   -n kube-system\r\n2018-04-05 14:31:32.349208 I | proto: duplicate proto type registered: google.protobuf.Any\r\n2018-04-05 14:31:32.349264 I | proto: duplicate proto type registered: google.protobuf.Duration\r\n2018-04-05 14:31:32.349278 I | proto: duplicate proto type registered: google.protobuf.Timestamp\r\nI0405 14:31:32.399235       1 factory.go:340] Creating scheduler from algorithm provider 'DefaultProvider'\r\nI0405 14:31:32.399249       1 factory.go:401] Creating scheduler with fit predicates 'map[NoVolumeNodeConflict:{} NoVolumeZoneConflict:{} MaxEBSVolumeCount:{} MaxAzureDiskVolumeCount:{} NoDiskConflict:{} PodToleratesNodeTaints:{} MaxGCEPDVolumeCount:{} MatchInterPodAffinity:{} GeneralPredicates:{} CheckNodeMemoryPressure:{} CheckNodeDiskPressure:{}]' and priority functions 'map[TaintTolerationPriority:{} SelectorSpreadPriority:{} InterPodAffinityPriority:{} LeastRequestedPriority:{} BalancedResourceAllocation:{} NodePreferAvoidPodsPriority:{} NodeAffinityPriority:{}]\r\nI0405 14:31:33.100049       1 controller_utils.go:994] Waiting for caches to sync for scheduler controller\r\nI0405 14:31:33.200162       1 controller_utils.go:1001] Caches are synced for scheduler controller\r\nI0405 14:31:33.200220       1 leaderelection.go:179] attempting to acquire leader lease...\r\nE0405 14:44:37.628519       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0405 15:14:18.516524       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0405 15:18:14.961635       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0405 17:16:49.757218       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0405 17:34:39.107423       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0405 19:03:40.206397       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0405 19:26:16.043213       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0405 19:27:49.275685       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0405 21:52:20.397328       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0405 23:24:05.957381       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0406 00:11:17.061948       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0406 02:04:41.839332       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0.0.1:2379 has no leader\r\nE0406 09:43:22.850567       1 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://127.0\r\n \r\nHere is  details etcd configuration:\r\nroot@k8s-master-3688E46E-0:/home/cineplanetadmin# etcd --version\r\netcd Version: 2.2.5\r\nGit SHA: Not provided (use ./build instead of go build)\r\nGo Version: go1.6.2\r\nGo OS/Arch: linux/amd64\r\nroot@k8s-master-3688E46E-0:/home/cineplanetadmin# sudo systemctl cat etcd\r\n# /lib/systemd/system/etcd.service\r\n[Unit]\r\nDescription=etcd - highly-available key value store\r\nDocumentation=https://github.com/coreos/etcd\r\nDocumentation=man:etcd\r\nAfter=network.target\r\nWants=network-online.target\r\n\r\n[Service]\r\nEnvironment=DAEMON_ARGS=\r\nEnvironment=ETCD_NAME=%H\r\nEnvironment=ETCD_DATA_DIR=/var/lib/etcd/default\r\nEnvironmentFile=-/etc/default/%p\r\nType=notify\r\nUser=etcd\r\nPermissionsStartOnly=true\r\n#ExecStart=/bin/sh -c \"GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS\"\r\nExecStart=/usr/bin/etcd $DAEMON_ARGS\r\nRestart=always\r\n#RestartSec=10s\r\nLimitNOFILE=65536\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\nAlias=etcd2.service\r\n\r\n\r\n#########################\r\n\r\n\r\nroot@k8s-master-3688E46E-1:/home/cineplanetadmin# etcd --version\r\netcd Version: 2.2.5\r\nGit SHA: Not provided (use ./build instead of go build)\r\nGo Version: go1.6.2\r\nGo OS/Arch: linux/amd64\r\nroot@k8s-master-3688E46E-1:/home/cineplanetadmin# sudo systemctl cat etcd\r\n# /lib/systemd/system/etcd.service\r\n[Unit]\r\nDescription=etcd - highly-available key value store\r\nDocumentation=https://github.com/coreos/etcd\r\nDocumentation=man:etcd\r\nAfter=network.target\r\nWants=network-online.target\r\n\r\n[Service]\r\nEnvironment=DAEMON_ARGS=\r\nEnvironment=ETCD_NAME=%H\r\nEnvironment=ETCD_DATA_DIR=/var/lib/etcd/default\r\nEnvironmentFile=-/etc/default/%p\r\nType=notify\r\nUser=etcd\r\nPermissionsStartOnly=true\r\n#ExecStart=/bin/sh -c \"GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS\"\r\nExecStart=/usr/bin/etcd $DAEMON_ARGS\r\nRestart=always\r\n#RestartSec=10s\r\nLimitNOFILE=65536\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\nAlias=etcd2.service\r\n\r\n\r\n\r\n####################\r\n\r\nroot@k8s-master-3688E46E-2:/home/cineplanetadmin# etcd --version\r\netcd Version: 2.2.5\r\nGit SHA: Not provided (use ./build instead of go build)\r\nGo Version: go1.6.2\r\nGo OS/Arch: linux/amd64\r\nroot@k8s-master-3688E46E-2:/home/cineplanetadmin# sudo systemctl cat etcd\r\n# /lib/systemd/system/etcd.service\r\n[Unit]\r\nDescription=etcd - highly-available key value store\r\nDocumentation=https://github.com/coreos/etcd\r\nDocumentation=man:etcd\r\nAfter=network.target\r\nWants=network-online.target\r\n\r\n[Service]\r\nEnvironment=DAEMON_ARGS=\r\nEnvironment=ETCD_NAME=%H\r\nEnvironment=ETCD_DATA_DIR=/var/lib/etcd/default\r\nEnvironmentFile=-/etc/default/%p\r\nType=notify\r\nUser=etcd\r\nPermissionsStartOnly=true\r\n#ExecStart=/bin/sh -c \"GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS\"\r\nExecStart=/usr/bin/etcd $DAEMON_ARGS\r\nRestart=always\r\n#RestartSec=10s\r\nLimitNOFILE=65536\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\nAlias=etcd2.service\r\n\r\n\r\ncommand service etcd status \r\n● etcd.service - etcd - highly-available key value store\r\n   Loaded: loaded (/lib/systemd/system/etcd.service; enabled; vendor preset: enabled)\r\n   Active: active (running) since Mon 2017-12-18 05:05:48 UTC; 4 months 10 days ago\r\n     Docs: https://github.com/coreos/etcd\r\n           man:etcd\r\n Main PID: 1224 (etcd)\r\n    Tasks: 13\r\n   Memory: 442.0M\r\n      CPU: 3d 4h 15min 14.923s\r\n   CGroup: /system.slice/etcd.service\r\n           └─1224 /usr/bin/etcd --name k8s-master-3688E46E-0 --initial-advertise-peer-urls http://10.240.255.5:2380 --listen-peer-urls http://10.240.255.5:2380 --advertise-client-urls http://10.240.255.5:2379 --listen-client-urls http://10.240.255.5:2379,http://127.0.0.1:2379 --initial-cluster-token k8s-etcd-cluster --initial-cluster k8s-master-3688E46E-0=http://10.240.255.5:2380,k8s-master-3688E46E-1=http://10.240.255.6:2380,k8s-master-3688E46E-2=http://10.240.255.7:2380 --data-dir /var/lib/etcddisk --initial-cluster-state new\r\n\r\nApr 29 22:09:33 k8s-master-3688E46E-0 etcd[1224]: 134af6e5d3e35861 [logterm: 12307, index: 153932328, vote: 0] rejected vote from 2a0965170762fd09 [logterm: 12307, index: 153932323] at term 12308\r\nApr 29 22:09:33 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 lost leader 134af6e5d3e35861 at term 12308\r\nApr 29 22:09:33 k8s-master-3688E46E-0 etcd[1224]: found conflict at index 153932324 [existing term: 12307, conflicting term: 12308]\r\nApr 29 22:09:33 k8s-master-3688E46E-0 etcd[1224]: replace the unstable entries from index 153932324\r\nApr 29 22:09:33 k8s-master-3688E46E-0 etcd[1224]: raft.node: 134af6e5d3e35861 elected leader 2a0965170762fd09 at term 12308\r\nApr 29 22:09:39 k8s-master-3688E46E-0 etcd[1224]: etcdserver: request timed out, possibly due to previous leader failure\r\nApr 29 22:15:00 k8s-master-3688E46E-0 etcd[1224]: start to snapshot (applied: 153936812, lastsnap: 153926811)\r\nApr 29 22:15:00 k8s-master-3688E46E-0 etcd[1224]: saved snapshot at index 153936812\r\nApr 29 22:15:00 k8s-master-3688E46E-0 etcd[1224]: compacted raft log at 153931812\r\nApr 29 22:15:27 k8s-master-3688E46E-0 etcd[1224]: purged file /var/lib/etcddisk/member/snap/0000000000003011-00000000092c2057.snap successfully\r\n\r\nAny ideas?\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/386113984","html_url":"https://github.com/coreos/etcd/issues/9668#issuecomment-386113984","issue_url":"https://api.github.com/repos/coreos/etcd/issues/9668","id":386113984,"user":{"login":"gyuho","id":6799218,"avatar_url":"https://avatars0.githubusercontent.com/u/6799218?v=4","gravatar_id":"","url":"https://api.github.com/users/gyuho","html_url":"https://github.com/gyuho","followers_url":"https://api.github.com/users/gyuho/followers","following_url":"https://api.github.com/users/gyuho/following{/other_user}","gists_url":"https://api.github.com/users/gyuho/gists{/gist_id}","starred_url":"https://api.github.com/users/gyuho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gyuho/subscriptions","organizations_url":"https://api.github.com/users/gyuho/orgs","repos_url":"https://api.github.com/users/gyuho/repos","events_url":"https://api.github.com/users/gyuho/events{/privacy}","received_events_url":"https://api.github.com/users/gyuho/received_events","type":"User","site_admin":false},"created_at":"2018-05-02T20:41:44Z","updated_at":"2018-05-02T20:41:44Z","author_association":"MEMBER","body":"> the service deployed will become very slow\r\n\r\nIf there's no leader, etcd service not only slows down, it will be unavailable."}}
{"actor":{"display_login":"gyuho"}}