{"push_id":3039077449,"size":3,"distinct_size":1,"ref":"refs/heads/no_shared_aws_vpc","head":"4ed6b1627cdbe2f400c890beaf4f9231c540ad6f","before":"3b46682328237ca49205ff1da9d034843701971a","commits":[{"sha":"109531f31cf4fd2a47f53fe27d3e707153913320","author":{"name":"W. Trevor King","email":"f75897670c17327c42ab201733fb8a81ff9f4c34@tremily.us"},"message":"cmd/openshift-install/create: Retry watch connections\n\nWe usually lose our watch when bootkube goes down.  From job [1]:\n\n  $ oc project ci-op-9g1vhqtz\n  $ oc logs -f --timestamps e2e-aws -c setup | tee /tmp/setup.log\n  ...\n  2018-11-03T04:19:55.121757935Z level=debug msg=\"added openshift-master-controllers.1563825412a4d77b: controller-manager-b5v49 became leader\"\n  ...\n  2018-11-03T04:20:14.679215171Z level=warning msg=\"RetryWatcher - getting event failed! Re-creating the watcher. Last RV: 3069\"\n  2018-11-03T04:20:16.539967372Z level=debug msg=\"added bootstrap-complete: cluster bootstrapping has completed\"\n  2018-11-03T04:20:16.540030121Z level=info msg=\"Destroying the bootstrap resources...\"\n  ...\n\nAnd simultaneously:\n\n  $ ssh -i libra.pem core@34.204.8.60 journalctl -f | tee /tmp/bootstrap.log\n  ...\n  Nov 03 04:20:14 ip-10-0-10-86 bootkube.sh[1033]: All self-hosted control plane components successfully started\n  Nov 03 04:20:14 ip-10-0-10-86 bootkube.sh[1033]: Tearing down temporary bootstrap control plane...\n  Nov 03 04:20:15 ip-10-0-10-86 hyperkube[840]: E1103 04:20:15.968877     840 kuberuntime_container.go:65] Can't make a ref to pod \"bootstrap-cluster-version-operator-ip-10-0-10-86_openshift-cluster-version(99ccfef8309f84bf88a0ca4a277097ac)\", container cluster-version-operator: selfLink was empty, can't make reference\n  Nov 03 04:20:15 ip-10-0-10-86 hyperkube[840]: E1103 04:20:15.975624     840 kuberuntime_container.go:65] Can't make a ref to pod \"bootstrap-kube-apiserver-ip-10-0-10-86_kube-system(427a4a342e137b5a9bb39a0feff24625)\", container kube-apiserver: selfLink was empty, can't make reference\n  Nov 03 04:20:16 ip-10-0-10-86 hyperkube[840]: W1103 04:20:16.002990     840 pod_container_deletor.go:75] Container \"0c647bcd6317ac2e06b625c44151aa6a3487aa1c47c5f1468213756f9a48ef91\" not found in pod's containers\n  Nov 03 04:20:16 ip-10-0-10-86 hyperkube[840]: W1103 04:20:16.005146     840 pod_container_deletor.go:75] Container \"bc47233151f1c0afaaee9e7abcfec9a515fe0a720ed2251fd7a51602c59060c5\" not found in pod's containers\n  Nov 03 04:20:16 ip-10-0-10-86 systemd[1]: progress.service holdoff time over, scheduling restart.\n  Nov 03 04:20:16 ip-10-0-10-86 systemd[1]: Starting Report the completion of the cluster bootstrap process...\n  Nov 03 04:20:16 ip-10-0-10-86 systemd[1]: Started Report the completion of the cluster bootstrap process.\n  Nov 03 04:20:16 ip-10-0-10-86 report-progress.sh[6828]: Reporting install progress...\n  Nov 03 04:20:16 ip-10-0-10-86 report-progress.sh[6828]: event/bootstrap-complete created\n  Nov 03 04:20:16 ip-10-0-10-86 hyperkube[840]: I1103 04:20:16.719526     840 reconciler.go:181] operationExecutor.UnmountVolume started for volume \"secrets\" (UniqueName: \"kubernetes.io/host-path/427a4a342e137b5a9bb39a0feff24625-secrets\") pod \"427a4a342e137b5a9bb39a0feff24625\" (UID: \"427a4a342e137b5a9bb39a0feff24625\")\n  ...\n\nThe resourceVersion watch will usually allow the re-created watcher to\npick up where its predecessor left off [2].  But in at least some\ncases, that doesn't seem to be happening [3]:\n\n  2018/11/02 23:30:00 Running pod e2e-aws\n  2018/11/02 23:48:00 Container test in pod e2e-aws completed successfully\n  2018/11/02 23:51:52 Container teardown in pod e2e-aws completed successfully\n  2018/11/03 00:08:33 Copying artifacts from e2e-aws into /logs/artifacts/e2e-aws\n  level=debug msg=\"Fetching \\\"Terraform Variables\\\"...\"\n  ...\n  level=debug msg=\"added openshift-master-controllers.1563734e367132e0: controller-manager-xlw62 became leader\"\n  level=warning msg=\"RetryWatcher - getting event failed! Re-creating the watcher. Last RV: 3288\"\n  level=fatal msg=\"Error executing openshift-install: waiting for bootstrap-complete: timed out waiting for the condition\"\n  2018/11/03 00:08:33 Container setup in pod e2e-aws failed, exit code 1, reason Error\n\nThat's unfortunately missing timestamps for the setup logs, but in the\nsuccessful logs from ci-op-9g1vhqtz, you can see the\ncontroller-manager becoming a leader around 20 seconds before\nbootstrap-complete.  That means the bootstrap-complete event probably\nfired around when the watcher dropped, which was probably well before\nthe setup container timed out (~38 minutes after it was launched).\nThe setup container timing out was probably the watch re-connect event\nhitting the 30 minute eventContext timeout.\n\nThe issue seems to be that when watcherFunc returns an error,\ndoReceive returned true, and receive exited, closing the doneChan (but\n*not* the resultChan).  That left UntilWithoutRetry hung waiting for a\nresult that never appeared (until the context timed it out).\n\nWith this commit, I'm retrying the watch conncetion every two seconds\nuntil we get a successful reconnect (or the context times out).  And\nI'm dropping the unused doneChan and closing resultChan instead to fix\nthe hang mentioned in the previous paragraph.\n\n[1]: https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_installer/595/pull-ci-openshift-installer-master-e2e-aws/1173\n[2]: https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#concurrency-control-and-consistency\n[3]: https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/45/pull-ci-openshift-cluster-version-operator-master-e2e-aws/58/build-log.txt","distinct":false,"url":"https://api.github.com/repos/staebler/installer/commits/109531f31cf4fd2a47f53fe27d3e707153913320"},{"sha":"09aa205d4df479476c06bc8d79e69091eaf7a13f","author":{"name":"OpenShift Merge Robot","email":"dd8cc73a9fcbc4f2f987fe818c87bd17f8347a2c@users.noreply.github.com"},"message":"Merge pull request #606 from wking/delay-watch-reconnect\n\ncmd/openshift-install/create: Retry watch connections","distinct":false,"url":"https://api.github.com/repos/staebler/installer/commits/09aa205d4df479476c06bc8d79e69091eaf7a13f"},{"sha":"4ed6b1627cdbe2f400c890beaf4f9231c540ad6f","author":{"name":"staebler","email":"b49b1ec815177ed068e3fd6cbfc8b45fbf28f4ea@redhat.com"},"message":"terraform/aws: remove option to use an existing vpc in aws\n\nFor the limited scope of the installer, we do not want the user to\nhave the ability to share the VPC between clusters. A shared VPC\ncould potentially be deleted when destroying one of the clusters,\nleaving the rest of the clusters using the shared VPC in an unusable\nstate.\n\nFixes https://jira.coreos.com/browse/CORS-873","distinct":true,"url":"https://api.github.com/repos/staebler/installer/commits/4ed6b1627cdbe2f400c890beaf4f9231c540ad6f"}]}
{"actor":{"display_login":"staebler"}}