{"action":"created","issue":{"url":"https://api.github.com/repos/openshift/openshift-ansible/issues/6916","repository_url":"https://api.github.com/repos/openshift/openshift-ansible","labels_url":"https://api.github.com/repos/openshift/openshift-ansible/issues/6916/labels{/name}","comments_url":"https://api.github.com/repos/openshift/openshift-ansible/issues/6916/comments","events_url":"https://api.github.com/repos/openshift/openshift-ansible/issues/6916/events","html_url":"https://github.com/openshift/openshift-ansible/pull/6916","id":292245544,"number":6916,"title":"Make openshift-ansible use static pods to install the control plane, make nodes prefer bootstrapping","user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars0.githubusercontent.com/u/1163175?v=4","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"labels":[{"id":681652779,"url":"https://api.github.com/repos/openshift/openshift-ansible/labels/size/XXL","name":"size/XXL","color":"ee0000","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2018-01-28T22:29:20Z","updated_at":"2018-02-21T05:12:50Z","closed_at":null,"author_association":"OWNER","pull_request":{"url":"https://api.github.com/repos/openshift/openshift-ansible/pulls/6916","html_url":"https://github.com/openshift/openshift-ansible/pull/6916","diff_url":"https://github.com/openshift/openshift-ansible/pull/6916.diff","patch_url":"https://github.com/openshift/openshift-ansible/pull/6916.patch"},"body":"1. Nodes continue to be configured for bootstrapping (as today)\r\n2. For bootstrap nodes, we write a generic bootstrap-node-config.yaml that contains static pod references and any bootstrap config, and then use that to start a child kubelet using `--write-flags` instead of launching the node ourselves.  If a node-config.yaml is laid down in `/etc/origin/node` it takes precedence.\r\n3. For 3.10 we want dynamic node config from Kubernetes to pull down additional files, but there are functional gaps.  For now, the openshift SDN container has a sidecar that syncs node config to disk and updates labels (kubelet doesn't update labels, https://github.com/kubernetes/kubernetes/issues/59314)\r\n4. On the masters, if openshift_master_bootstrap_enabled we generate the master-config.yaml and the etcd config, but we don't start etcd or the masters (no services installed)\r\n5. On the masters, we copy the static files into the correct pod-manifest-path (/etc/origin/node/pods) or similar\r\n6. The kubelet at that point should automatically pick up the new static files and launch the components\r\n7. We wait for them to converge\r\n8. We install openshift-sdn as the first component, which allows nodes to go ready and start installing things.  There is a gap here where the masters are up, the nodes can bootstrap, but the nodes are not ready because no network plugin is installed.\r\n\r\nChallenges at this point:\r\n\r\n* The master shims (`master-logs` and `master-restart`) need to deal with CRI-O and systemd.  Ideally this is a temporary shim until we remove systemd for these components and have cri-ctl installed.\r\n* We need to test failure modes of the static pods\r\n* Testing\r\n\r\nFurther exploration things:\r\n\r\n* need to get all the images using image streams or properly replaced into the static pods\r\n* need to look at upgrades and updates\r\n* disk locations become our API (`/var/lib/origin`, `/var/lib/etcd`) - how many customers have fiddled with this?\r\n* may need to make the kubelet halt if it hasn't been able to get server/client certs within a bounded window (5m?) so to ensure that autoheals happen (https://github.com/openshift/origin/pull/18430)\r\n* have to figure out whether dynamic kubelet config is a thing we can rely on for 3.10 (@liggitt), and what gaps there are with dynamic reconfig\r\n* client-ca.crt is not handled by bootstrapping or dynamic config.  This needs a solution unless we keep the openshift-sdn sidecar around\r\n* kubelet doesn't send sd notify to systemd (https://github.com/kubernetes/kubernetes/issues/59079)\r\n\r\n@derekwaynecarr @sdodson @liggitt @deads2k this is the core of self-hosting."},"comment":{"url":"https://api.github.com/repos/openshift/openshift-ansible/issues/comments/367215851","html_url":"https://github.com/openshift/openshift-ansible/pull/6916#issuecomment-367215851","issue_url":"https://api.github.com/repos/openshift/openshift-ansible/issues/6916","id":367215851,"user":{"login":"openshift-ci-robot","id":30189103,"avatar_url":"https://avatars1.githubusercontent.com/u/30189103?v=4","gravatar_id":"","url":"https://api.github.com/users/openshift-ci-robot","html_url":"https://github.com/openshift-ci-robot","followers_url":"https://api.github.com/users/openshift-ci-robot/followers","following_url":"https://api.github.com/users/openshift-ci-robot/following{/other_user}","gists_url":"https://api.github.com/users/openshift-ci-robot/gists{/gist_id}","starred_url":"https://api.github.com/users/openshift-ci-robot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/openshift-ci-robot/subscriptions","organizations_url":"https://api.github.com/users/openshift-ci-robot/orgs","repos_url":"https://api.github.com/users/openshift-ci-robot/repos","events_url":"https://api.github.com/users/openshift-ci-robot/events{/privacy}","received_events_url":"https://api.github.com/users/openshift-ci-robot/received_events","type":"User","site_admin":false},"created_at":"2018-02-21T05:12:50Z","updated_at":"2018-02-21T05:12:50Z","author_association":"NONE","body":"@smarterclayton: The following tests **failed**, say `/retest` to rerun them all:\n\nTest name | Commit | Details | Rerun command\n--- | --- | --- | ---\nci/openshift-jenkins/gcp | cb532bd37153ad0fdae0bcabbf305e7d6e44dfe9 | [link](https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_openshift-ansible/6916/test_pull_request_openshift_ansible_extended_conformance_gce/990/) | `/test gcp`\nci/openshift-jenkins/system-containers | cb532bd37153ad0fdae0bcabbf305e7d6e44dfe9 | [link](https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_openshift-ansible/6916/test_pull_request_openshift_ansible_extended_conformance_install_system_containers/5149/) | `/test system-containers`\nci/openshift-jenkins/extended_conformance_install_crio | cb532bd37153ad0fdae0bcabbf305e7d6e44dfe9 | [link](https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_openshift-ansible/6916/test_pull_request_openshift_ansible_extended_conformance_install_crio/2528/) | `/test crio`\nci/openshift-jenkins/install | cb532bd37153ad0fdae0bcabbf305e7d6e44dfe9 | [link](https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_openshift-ansible/6916/test_pull_request_openshift_ansible_extended_conformance_install/5434/) | `/test install`\nci/openshift-jenkins/logging | a0ece819220e24b0b4cdf83b749b6926e84bf4da | [link](https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_openshift-ansible/6916/test_pull_request_openshift_ansible_logging/6112/) | `/test logging`\n\n[Full PR test history](https://openshift-gce-devel.appspot.com/pr/openshift_openshift-ansible/6916). [Your PR dashboard](https://openshift-gce-devel.appspot.com/pr/smarterclayton). Please help us cut down on flakes by [linking to](https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/openshift/openshift-ansible/issues?q=is:issue+is:open) when you hit one in your PR.\n\n<details>\n\nInstructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/devel/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).\n</details>\n<!-- test report -->"}}
{"actor":{"display_login":"openshift-ci-robot"}}