{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kops/issues/1917","repository_url":"https://api.github.com/repos/kubernetes/kops","labels_url":"https://api.github.com/repos/kubernetes/kops/issues/1917/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kops/issues/1917/comments","events_url":"https://api.github.com/repos/kubernetes/kops/issues/1917/events","html_url":"https://github.com/kubernetes/kops/issues/1917","id":207920265,"node_id":"MDU6SXNzdWUyMDc5MjAyNjU=","number":1917,"title":"kops update cluster stuck on nat gateway update","user":{"login":"hollowimage","id":7103716,"node_id":"MDQ6VXNlcjcxMDM3MTY=","avatar_url":"https://avatars2.githubusercontent.com/u/7103716?v=4","gravatar_id":"","url":"https://api.github.com/users/hollowimage","html_url":"https://github.com/hollowimage","followers_url":"https://api.github.com/users/hollowimage/followers","following_url":"https://api.github.com/users/hollowimage/following{/other_user}","gists_url":"https://api.github.com/users/hollowimage/gists{/gist_id}","starred_url":"https://api.github.com/users/hollowimage/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hollowimage/subscriptions","organizations_url":"https://api.github.com/users/hollowimage/orgs","repos_url":"https://api.github.com/users/hollowimage/repos","events_url":"https://api.github.com/users/hollowimage/events{/privacy}","received_events_url":"https://api.github.com/users/hollowimage/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2017-02-15T20:35:43Z","updated_at":"2018-06-02T21:13:18Z","closed_at":null,"author_association":"NONE","body":"so im hitting a weird issue with using kops to do any sort of cluster updates post creation\r\n\r\n```\r\n[15:26:36][admin@domain.com]$ kops update cluster\r\nUsing cluster from kubectl context: prod-domain.com\r\n\r\nI0215 15:26:55.140339   24888 dns.go:89] Private DNS: skipping DNS validation\r\nI0215 15:26:55.147283   24888 executor.go:91] Tasks: 0 done / 96 total; 31 can run\r\nI0215 15:27:01.612480   24888 executor.go:91] Tasks: 31 done / 96 total; 21 can run\r\nI0215 15:27:02.418030   24888 executor.go:91] Tasks: 52 done / 96 total; 30 can run\r\nI0215 15:27:09.089526   24888 executor.go:91] Tasks: 82 done / 96 total; 8 can run\r\nI0215 15:27:15.795805   24888 dnsname.go:107] AliasTarget for \"api.prod-domain.com.\" is \"prod-domain.us-east-1.elb.amazonaws.com.\"\r\nW0215 15:27:15.903281   24888 executor.go:109] error running task \"NatGateway/us-east-1dprod-domain.com\" (9m53s remaining to succeed): Field cannot be changed: ElasticIp\r\nW0215 15:27:15.903313   24888 executor.go:109] error running task \"NatGateway/us-east-1c.prod-domain.com\" (9m53s remaining to succeed): Field cannot be changed: ElasticIp\r\nW0215 15:27:15.903324   24888 executor.go:109] error running task \"NatGateway/us-east-1b.prod-domain.com\" (9m53s remaining to succeed): Field cannot be changed: ElasticIp\r\nI0215 15:27:15.903342   24888 executor.go:91] Tasks: 87 done / 96 total; 6 can run\r\nW0215 15:27:16.040721   24888 executor.go:109] error running task \"NatGateway/us-east-1d.prod-domain.com\" (9m53s remaining to succeed): Field cannot be changed: ElasticIp\r\nW0215 15:27:16.040746   24888 executor.go:109] error running task \"NatGateway/us-east-1c.prod-domain.com\" (9m53s remaining to succeed): Field cannot be changed: ElasticIp\r\nW0215 15:27:16.040757   24888 executor.go:109] error running task \"NatGateway/us-east-1b.prod-domain.com\" (9m53s remaining to succeed): Field cannot be changed: ElasticIp\r\nI0215 15:27:16.040774   24888 executor.go:91] Tasks: 90 done / 96 total; 3 can run\r\nW0215 15:27:16.141043   24888 executor.go:109] error running task \"NatGateway/us-east-1d.prod-domain.com\" (9m52s remaining to succeed): Field cannot be changed: ElasticIp\r\nW0215 15:27:16.141069   24888 executor.go:109] error running task \"NatGateway/us-east-1c.prod-domain.com\" (9m52s remaining to succeed): Field cannot be changed: ElasticIp\r\nW0215 15:27:16.141079   24888 executor.go:109] error running task \"NatGateway/us-east-1b.prod-domain.com\" (9m52s remaining to succeed): Field cannot be changed: ElasticIp\r\nI0215 15:27:16.141089   24888 executor.go:124] No progress made, sleeping before retrying 3 failed task(s)\r\n```\r\n\r\ni am using `Version 1.5.1 (git-01deca8)` and\r\n\r\n```\r\n$ kubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"5\", GitVersion:\"v1.5.2\", GitCommit:\"08e099554f3c31f6e6f07b448ab3ed78d0520507\", GitTreeState:\"clean\", BuildDate:\"2017-01-12T04:57:25Z\", GoVersion:\"go1.7.4\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"5\", GitVersion:\"v1.5.2\", GitCommit:\"08e099554f3c31f6e6f07b448ab3ed78d0520507\", GitTreeState:\"clean\", BuildDate:\"2017-01-12T04:52:34Z\", GoVersion:\"go1.7.4\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\ncreation summary:\r\n1. using a shared vpc\r\n2. everything else was created by kops\r\n3. private cluster topology with weave\r\n\r\n```\r\nkops create cluster --cloud aws --name prod.domain.com --dns private --dns-zone [ZONE] --vpc=vpc-xxxxxx --topology private --master-size=m4.large --master-zones=us-east-1b,us-east-1c,us-east-1d --network-cidr =172.x.x.x/16 --node-count=3 --node-size=m4.large --zones=us-east-1b,us-east-1c,us-east-1d --networking weave --channel alpha --associate-public-ip=false\r\n```\r\nI made the following changes to the cluster post-initial creation:\r\n\r\n1. change egress in the route tables for utility subnets to point to custom firewall solution we have in place\r\n2. change security group rules to allow inbound traffic to the public subnets from `0.0.0.0/0` to `x.x.x.x` where the latter is the proxy source IP that handles our VPC egress and ingress\r\n3. i left NGWs it created with their EIPs as is. nothing was detached or deleted. \r\n\r\nin summary the only real change was the subnet egress routing and the security group for inbound traffic to block public access. \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kops/issues/comments/394117508","html_url":"https://github.com/kubernetes/kops/issues/1917#issuecomment-394117508","issue_url":"https://api.github.com/repos/kubernetes/kops/issues/1917","id":394117508,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NDExNzUwOA==","user":{"login":"fejta-bot","id":30488982,"node_id":"MDQ6VXNlcjMwNDg4OTgy","avatar_url":"https://avatars1.githubusercontent.com/u/30488982?v=4","gravatar_id":"","url":"https://api.github.com/users/fejta-bot","html_url":"https://github.com/fejta-bot","followers_url":"https://api.github.com/users/fejta-bot/followers","following_url":"https://api.github.com/users/fejta-bot/following{/other_user}","gists_url":"https://api.github.com/users/fejta-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/fejta-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fejta-bot/subscriptions","organizations_url":"https://api.github.com/users/fejta-bot/orgs","repos_url":"https://api.github.com/users/fejta-bot/repos","events_url":"https://api.github.com/users/fejta-bot/events{/privacy}","received_events_url":"https://api.github.com/users/fejta-bot/received_events","type":"User","site_admin":false},"created_at":"2018-06-02T21:13:18Z","updated_at":"2018-06-02T21:13:18Z","author_association":"NONE","body":"Issues go stale after 90d of inactivity.\nMark the issue as fresh with `/remove-lifecycle stale`.\nStale issues rot after an additional 30d of inactivity and eventually close.\n\nIf this issue is safe to close now please do so with `/close`.\n\nSend feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta).\n/lifecycle stale"}}
{"actor":{"display_login":"fejta-bot"}}