{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66770","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66770/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66770/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66770/events","html_url":"https://github.com/kubernetes/kubernetes/issues/66770","id":345740935,"node_id":"MDU6SXNzdWUzNDU3NDA5MzU=","number":66770,"title":"Failing Test : [sig-api-machinary] Etcd failure [Disruptive]","user":{"login":"mohammedzee1000","id":8547729,"node_id":"MDQ6VXNlcjg1NDc3Mjk=","avatar_url":"https://avatars3.githubusercontent.com/u/8547729?v=4","gravatar_id":"","url":"https://api.github.com/users/mohammedzee1000","html_url":"https://github.com/mohammedzee1000","followers_url":"https://api.github.com/users/mohammedzee1000/followers","following_url":"https://api.github.com/users/mohammedzee1000/following{/other_user}","gists_url":"https://api.github.com/users/mohammedzee1000/gists{/gist_id}","starred_url":"https://api.github.com/users/mohammedzee1000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mohammedzee1000/subscriptions","organizations_url":"https://api.github.com/users/mohammedzee1000/orgs","repos_url":"https://api.github.com/users/mohammedzee1000/repos","events_url":"https://api.github.com/users/mohammedzee1000/events{/privacy}","received_events_url":"https://api.github.com/users/mohammedzee1000/received_events","type":"User","site_admin":false},"labels":[{"id":496752468,"node_id":"MDU6TGFiZWw0OTY3NTI0Njg=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/failing-test","name":"kind/failing-test","color":"e11d21","default":false},{"id":687345274,"node_id":"MDU6TGFiZWw2ODczNDUyNzQ=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/milestone/removed","name":"milestone/removed","color":"ededed","default":false},{"id":114528223,"node_id":"MDU6TGFiZWwxMTQ1MjgyMjM=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/important-soon","name":"priority/important-soon","color":"eb6420","default":false},{"id":173493835,"node_id":"MDU6TGFiZWwxNzM0OTM4MzU=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/api-machinery","name":"sig/api-machinery","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-07-30T12:41:50Z","updated_at":"2018-08-10T03:09:10Z","closed_at":null,"author_association":"MEMBER","body":"# Failing Jobs\r\n[sig-release-master-upgrade#gce-1.10-master-upgrade](https://k8s-testgrid.appspot.com/sig-release-master-upgrade#gce-1.10-master-upgrade-master)\r\n\r\n# Failing Tests\r\n[[sig-api-machinery] Etcd failure [Disruptive] should recover from SIGKILL](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1668)\r\n\r\n/sig api-machinery\r\n/kind failing-test\r\n/priority important-soon\r\n/milestone v1.12"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/411962392","html_url":"https://github.com/kubernetes/kubernetes/issues/66770#issuecomment-411962392","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/66770","id":411962392,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTk2MjM5Mg==","user":{"login":"roycaihw","id":6629765,"node_id":"MDQ6VXNlcjY2Mjk3NjU=","avatar_url":"https://avatars1.githubusercontent.com/u/6629765?v=4","gravatar_id":"","url":"https://api.github.com/users/roycaihw","html_url":"https://github.com/roycaihw","followers_url":"https://api.github.com/users/roycaihw/followers","following_url":"https://api.github.com/users/roycaihw/following{/other_user}","gists_url":"https://api.github.com/users/roycaihw/gists{/gist_id}","starred_url":"https://api.github.com/users/roycaihw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/roycaihw/subscriptions","organizations_url":"https://api.github.com/users/roycaihw/orgs","repos_url":"https://api.github.com/users/roycaihw/repos","events_url":"https://api.github.com/users/roycaihw/events{/privacy}","received_events_url":"https://api.github.com/users/roycaihw/received_events","type":"User","site_admin":false},"created_at":"2018-08-10T03:09:10Z","updated_at":"2018-08-10T03:09:10Z","author_association":"MEMBER","body":"etcd is taking >30s to recover, which already exceeded the [sleeping time](https://github.com/kubernetes/kubernetes/blob/a1b3133b4648ab804280a4c26ee1c5015c708d19/test/e2e/apimachinery/etcd_failure.go#L86) (20s) in the [e2e test](https://github.com/kubernetes/kubernetes/blob/a1b3133b4648ab804280a4c26ee1c5015c708d19/test/e2e/apimachinery/etcd_failure.go#L64-L70)\r\n\r\nhttps://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1692\r\netcd took 35s to recover\r\n```\r\n(etcd last previous log was at 2018-08-05 05:47:32.045810 I | mvcc: finished scheduled compaction at 76335 (took 2.141101ms))\r\n\r\nE2E test killed etcd and then slept for 20s\r\n05:49:54.019] failing etcd\r\n\r\nE2E test querying apiserver\r\n05:50:14.311 deleting pods from existing replication controller\r\n\r\nthe last apiserver healthz check failed before apiserver gets killed\r\n05:50:25.277929       1 wrap.go:42] GET /healthz: (3.442000165s) 500\r\n\r\nkubelet decided to kill apiserver\r\n05:50:27.998952    1508 kuberuntime_manager.go:549] Container \"kube-apiserver\" ({\"docker\" \"b5f2b16d8d798509c1fb04e7631be770e657648733dac943d164455a511bef31\"}) of pod kube-apiserver-bootstrap-e2e-master_kube-system(d90dc4fdfd9719be8e1316f25d4a2530): Container failed liveness probe.. Container will be killed and recreated.\r\n\r\netcd restarted\r\n05:50:29.143751 I | pkg/flags: recognized and used environment variable ETCD_SNAPSHOT_COUNT=10000\r\n\r\nE2E test failed and teared down test namespace\r\n05:50:35.213] [AfterEach] [sig-api-machinery] Etcd failure [Disruptive]\r\n\r\napiserver restarts\r\n05:50:36.177053       1 flags.go:27] FLAG: --address=\"127.0.0.1\"\r\n```\r\n\r\nhttps://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1693\r\netcd took 31s to recover\r\n```\r\n(etcd last previous log was at 2018-08-05 21:41:41.326661 I | mvcc: finished scheduled compaction at 113290 (took 10.066944ms))\r\n\r\nE2E test killed etcd and then slept for 20s\r\n21:43:08.435 failing etcd\r\n\r\nE2E test querying apiserver\r\n21:43:28.699 deleting pods from existing replication controller\r\n\r\netcd restarted\r\n21:43:39.686431 I | pkg/flags: recognized and used environment variable ETCD_SNAPSHOT_COUNT=10000\r\n\r\nthe last apiserver healthz check failed before apiserver gets killed\r\n21:43:42.950523       1 wrap.go:42] GET /healthz: (3.820478578s) 500\r\n\r\nkubelet decided to kill apiserver\r\n21:43:42.957716    1500 kuberuntime_manager.go:549] Container \"kube-apiserver\" ({\"docker\" \"a6a95cdd07858c5348bbb511cc26a323abd3d45c9e681b45916700b175320153\"}) of pod kube-apiserver-bootstrap-e2e-master_kube-system(d90dc4fdfd9719be8e1316f25d4a2530): Container failed liveness probe.. Container will be killed and recreated.\r\n\r\nE2E test failed and teared down test namespace\r\n21:43:46.135] [AfterEach] [sig-api-machinery] Etcd failure [Disruptive]\r\n\r\napiserver restarts\r\n21:43:48.009834       1 flags.go:27] FLAG: --address=\"127.0.0.1\"\r\n```"}}
{"actor":{"display_login":"roycaihw"}}