{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry-incubator/kubo-deployment/issues/344","repository_url":"https://api.github.com/repos/cloudfoundry-incubator/kubo-deployment","labels_url":"https://api.github.com/repos/cloudfoundry-incubator/kubo-deployment/issues/344/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry-incubator/kubo-deployment/issues/344/comments","events_url":"https://api.github.com/repos/cloudfoundry-incubator/kubo-deployment/issues/344/events","html_url":"https://github.com/cloudfoundry-incubator/kubo-deployment/issues/344","id":360719391,"node_id":"MDU6SXNzdWUzNjA3MTkzOTE=","number":344,"title":"Adding/removing a master node to the cluster is kicking off a whole cluster restart (canary deployment)","user":{"login":"lgunta2018","id":43278032,"node_id":"MDQ6VXNlcjQzMjc4MDMy","avatar_url":"https://avatars1.githubusercontent.com/u/43278032?v=4","gravatar_id":"","url":"https://api.github.com/users/lgunta2018","html_url":"https://github.com/lgunta2018","followers_url":"https://api.github.com/users/lgunta2018/followers","following_url":"https://api.github.com/users/lgunta2018/following{/other_user}","gists_url":"https://api.github.com/users/lgunta2018/gists{/gist_id}","starred_url":"https://api.github.com/users/lgunta2018/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lgunta2018/subscriptions","organizations_url":"https://api.github.com/users/lgunta2018/orgs","repos_url":"https://api.github.com/users/lgunta2018/repos","events_url":"https://api.github.com/users/lgunta2018/events{/privacy}","received_events_url":"https://api.github.com/users/lgunta2018/received_events","type":"User","site_admin":false},"labels":[{"id":539523607,"node_id":"MDU6TGFiZWw1Mzk1MjM2MDc=","url":"https://api.github.com/repos/cloudfoundry-incubator/kubo-deployment/labels/unscheduled","name":"unscheduled","color":"D9D9D9","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-09-17T04:26:31Z","updated_at":"2018-09-19T20:14:17Z","closed_at":null,"author_association":"NONE","body":"<!--\r\nThis form is for bug reports ONLY!\r\n\r\nIf you're looking for Kubernetes-specific help check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes)\r\nand the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).\r\n-->\r\n\r\n**What happened**:\r\nAdding/removing a master node to the cluster is kicking off the whole cluster restart (canary deployment)\r\n\r\n**What you expected to happen**:\r\nAdding/removing a master node should not kick off the whole cluster restart\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n\r\n Steps: \r\n1. Create a cluster with 3 master and 3 worker nodes\r\n2. Update the number of master nodes instances to 2 in cfcr.yml file\r\n3. Run the bosh deployment command, you can see the whole cluster restart \r\n\r\nrunlog:\r\n=======\r\nRemoving master node:\r\n-----------------------\r\nbosh deploy -d cfcr ${KD}/manifests/cfcr.yml -o ${KD}/manifests/ops-files/iaas/aws/cloud-provider.yml -o cfcr-ops.yml -l <(bbl outputs)\r\nUsing environment 'https://10.0.0.6:25555' as client 'admin'\r\n\r\nUsing deployment 'cfcr'\r\n\r\nRelease 'cfcr-etcd/1.5.0' already exists.\r\n\r\nRelease 'bpm/0.12.3' already exists.\r\n\r\naddons:\r\n- include:\r\n    stemcells:\r\n    - os: ubuntu-trusty\r\n    - os: ubuntu-xenial\r\n  name: bosh-dns\r\n- jobs:\r\n  - name: kubo-dns-aliases\r\n    release: kubo\r\n  name: bosh-dns-aliases\r\nfeatures:\r\n  use_dns_addresses: true\r\ninstance_groups:\r\n- azs:\r\n  - z1\r\n  instances: 1\r\n  jobs:\r\n  - name: apply-specs\r\n    properties:\r\n      addons:\r\n      - kube-dns\r\n      - metrics-server\r\n      - heapster\r\n      - kubernetes-dashboard\r\n      admin-password: ((kubo-admin-password))\r\n      admin-username: admin\r\n      api-token: ((kubelet-password))\r\n      tls:\r\n        heapster: ((tls-heapster))\r\n        influxdb: ((tls-influxdb))\r\n        kubernetes: ((tls-kubernetes))\r\n        kubernetes-dashboard: ((tls-kubernetes-dashboard))\r\n        metrics-server: ((tls-metrics-server))\r\n    release: kubo\r\n  lifecycle: errand\r\n  name: apply-addons\r\n  networks:\r\n  - name: default\r\n  stemcell: default\r\n\"kubo-deployment/manifests/cfcr.yml\" 297L, 7524C\r\n  type: certificate\r\n- name: tls-etcdctl\r\n  options:\r\n    ca: kubo_ca\r\n    common_name: etcdClient\r\n    extended_key_usage:\r\n    - client_auth\r\n  type: certificate\r\n- name: tls-metrics-server\r\n  options:\r\n    alternative_names:\r\n    - metrics-server.kube-system.svc\r\n    ca: kubo_ca\r\n    common_name: metrics-server\r\n  type: certificate\r\n- name: tls-heapster\r\n  options:\r\n    alternative_names:\r\n    - heapster.kube-system.svc.cluster.local\r\n    ca: kubo_ca\r\n    common_name: heapster\r\n  type: certificate\r\n- name: tls-influxdb\r\n  options:\r\n    alternative_names: []\r\n    ca: kubo_ca\r\n    common_name: monitoring-influxdb\r\n  type: certificate\r\n- name: kubernetes-dashboard-ca\r\n  options:\r\n    common_name: ca\r\n    is_ca: true\r\n  type: certificate\r\n- name: tls-kubernetes-dashboard\r\n  options:\r\n    alternative_names: []\r\n    ca: kubernetes-dashboard-ca\r\n    common_name: kubernetesdashboard.cfcr.internal\r\n  type: certificate\r\n?instances\r\n          certificate: ((tls-etcdctl.certificate))\r\nRelease 'bosh-dns/1.8.0' already exists.\r\n\r\nRelease 'docker/32.0.0' already exists.\r\n\r\n  instance_groups:\r\n  - name: master\r\n-   instances: 3\r\n+   instances: 2\r\n\r\nContinue? [yN]: y\r\n\r\nTask 120\r\n\r\nTask 120 | 23:47:07 | Preparing deployment: Preparing deployment (00:00:07)\r\nTask 120 | 23:47:39 | Preparing package compilation: Finding packages to compile (00:00:00)\r\nTask 120 | 23:47:39 | Deleting unneeded instances master: master/a02c36fb-a6d0-4ca9-8178-8a929087d32e (2) (00:00:43)\r\nTask 120 | 23:48:22 | Updating instance master: master/ba1fbbda-5080-4094-b8c8-671d9abb34a6 (0) (canary) (00:01:05)\r\nTask 120 | 23:49:27 | Updating instance master: master/c5dd8b42-ee9e-4607-ad44-152144a7eebf (1) (00:01:22)\r\nTask 120 | 23:50:49 | Updating instance worker: worker/43ac3278-5a5b-4a9b-a782-e3b52254f98d (0) (canary) (00:00:33)\r\nTask 120 | 23:51:22 | Updating instance worker: worker/41823365-c567-4acc-a2c3-d3df897ee8b3 (1) (00:00:35)\r\nTask 120 | 23:51:57 | Updating instance worker: worker/598eec97-2305-4eac-a784-2fee04d6121b (2) (00:00:41)\r\n\r\nSame behavior is observed for adding a new master node:\r\n==============================\r\n\r\n→ bosh deploy -d cfcr ${KD}/manifests/cfcr.yml -o ${KD}/manifests/ops-files/iaas/aws/cloud-provider.yml -o cfcr-ops.yml -l <(bbl outputs)\r\nUsing environment 'https://10.0.0.6:25555' as client 'admin'\r\n\r\nUsing deployment 'cfcr'\r\n\r\nRelease 'bpm/0.12.3' already exists.\r\n\r\nRelease 'docker/32.0.0' already exists.\r\n\r\nRelease 'cfcr-etcd/1.5.0' already exists.\r\n\r\nRelease 'bosh-dns/1.8.0' already exists.\r\n\r\n  instance_groups:\r\n  - name: master\r\n-   instances: 2\r\n+   instances: 3\r\n\r\nContinue? [yN]: y\r\n\r\nTask 234\r\n\r\nTask 234 | 00:10:30 | Preparing deployment: Preparing deployment (00:00:06)\r\nTask 234 | 00:11:07 | Preparing package compilation: Finding packages to compile (00:00:00)\r\nTask 234 | 00:11:07 | Creating missing vms: master/09e9e7e6-7e1f-4d51-b591-1c0f1e8ca12d (2) (00:01:27)\r\nTask 234 | 00:12:34 | Updating instance master: master/ba1fbbda-5080-4094-b8c8-671d9abb34a6 (0) (canary) (00:01:07)\r\nTask 234 | 00:13:41 | Updating instance master: master/c5dd8b42-ee9e-4607-ad44-152144a7eebf (1) (00:01:05)\r\nTask 234 | 00:14:46 | Updating instance master: master/09e9e7e6-7e1f-4d51-b591-1c0f1e8ca12d (2) (00:01:27)\r\nTask 234 | 00:16:13 | Updating instance worker: worker/43ac3278-5a5b-4a9b-a782-e3b52254f98d (0) (canary) (00:00:33)\r\nTask 234 | 00:16:46 | Updating instance worker: worker/41823365-c567-4acc-a2c3-d3df897ee8b3 (1) (00:00:42)\r\nTask 234 | 00:17:28 | Updating instance worker: worker/598eec97-2305-4eac-a784-2fee04d6121b (2) (00:00:35)\r\n\r\nTask 234 Started  Sat Sep 15 00:10:30 UTC 2018\r\nTask 234 Finished Sat Sep 15 00:18:03 UTC 2018\r\nTask 234 Duration 00:07:33\r\nTask 234 done\r\n\r\n**Anything else we need to know?**:\r\nAdding worker nodes is working fine, it is not restarting the whole cluster.\r\nkube-deployment : v0.21.0\r\n\r\n**Environment**:\r\n- Deployment Info (`bosh -d <deployment> deployment`):\r\nbosh deploy -d cfcr ${KD}/manifests/cfcr.yml -o ${KD}/manifests/ops-files/iaas/aws/cloud-provider.yml -o cfcr-ops.yml -l <(bbl outputs)\r\n\r\name  Release(s)       Stemcell(s)                                    Config(s)        Team(s)\r\ncfcr  bosh-dns/1.8.0   bosh-aws-xen-hvm-ubuntu-xenial-go_agent/97.16  1 cloud/default  -\r\n      bpm/0.12.3                                                      2 runtime/dns\r\n      cfcr-etcd/1.5.0\r\n      docker/32.0.0\r\n      kubo/0.21.0\r\n\r\n- Environment Info (`bosh -e <environment> environment`):\r\nName      bosh-cfcr-lgunta\r\nUUID      58cd4562-b14b-4310-aaa3-bd1582250f34\r\nVersion   267.5.0 (00000000)\r\nCPI       aws_cpi\r\nFeatures  compiled_package_cache: disabled\r\n          config_server: enabled\r\n          dns: disabled\r\n          snapshots: disabled\r\nUser      admin\r\n\r\n- Kubernetes version (`kubectl version`):\r\n\r\nClient Version: version.Info{Major:\"1\", Minor:\"11\", GitVersion:\"v1.11.3\", GitCommit:\"a4529464e4629c21224b3d52edfe0ea91b072862\", GitTreeState:\"clean\", BuildDate:\"2018-09-10T11:44:36Z\", GoVersion:\"go1.11\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"11\", GitVersion:\"v1.11.2\", GitCommit:\"bb9ffb1654d4a729bb4cec18ff088eacc153c239\", GitTreeState:\"clean\", BuildDate:\"2018-08-07T23:08:19Z\", GoVersion:\"go1.10.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n\r\n- Cloud provider (e.g. `aws`, `gcp`, `vsphere`):\r\nAWS"},"comment":{"url":"https://api.github.com/repos/cloudfoundry-incubator/kubo-deployment/issues/comments/422942603","html_url":"https://github.com/cloudfoundry-incubator/kubo-deployment/issues/344#issuecomment-422942603","issue_url":"https://api.github.com/repos/cloudfoundry-incubator/kubo-deployment/issues/344","id":422942603,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMjk0MjYwMw==","user":{"login":"alex-slynko","id":4385389,"node_id":"MDQ6VXNlcjQzODUzODk=","avatar_url":"https://avatars3.githubusercontent.com/u/4385389?v=4","gravatar_id":"","url":"https://api.github.com/users/alex-slynko","html_url":"https://github.com/alex-slynko","followers_url":"https://api.github.com/users/alex-slynko/followers","following_url":"https://api.github.com/users/alex-slynko/following{/other_user}","gists_url":"https://api.github.com/users/alex-slynko/gists{/gist_id}","starred_url":"https://api.github.com/users/alex-slynko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alex-slynko/subscriptions","organizations_url":"https://api.github.com/users/alex-slynko/orgs","repos_url":"https://api.github.com/users/alex-slynko/repos","events_url":"https://api.github.com/users/alex-slynko/events{/privacy}","received_events_url":"https://api.github.com/users/alex-slynko/received_events","type":"User","site_admin":false},"created_at":"2018-09-19T20:14:17Z","updated_at":"2018-09-19T20:14:17Z","author_association":"MEMBER","body":"Hi @lgunta2018 \r\n\r\n1. If you want to test how the cluster react to removing master node, you need to use `delete-vm` or `stop` command. I wrote[ a tiny blog post](https://medium.com/@slynko/deleting-virtual-machines-using-bosh-80826ce4163e) where I tried to explain the difference.\r\n2. We created[ a spike](https://www.pivotaltracker.com/story/show/160632229) to investigate it more, but I can't guarantee we will work on it or fix it soon. It might require big redesign for CFCR or some architectural Bosh changes. We might prioritize it if we see a business value in improving it.\r\n\r\nThere are two workarounds for this I can think of, but we haven't tried them \r\n - create external load balancer and use [manual links](https://bosh.io/docs/links-manual/) to pass traffic through this load balancer.\r\n- split deployments into two: control plane and workers and use [cross-deployment links ](https://bosh.io/docs/links/#cross-deployment)\r\n\r\nFeel free to ask more questions in Slack channel or here."}}
{"actor":{"display_login":"alex-slynko"}}