{"action":"created","issue":{"url":"https://api.github.com/repos/terraform-providers/terraform-provider-google/issues/1356","repository_url":"https://api.github.com/repos/terraform-providers/terraform-provider-google","labels_url":"https://api.github.com/repos/terraform-providers/terraform-provider-google/issues/1356/labels{/name}","comments_url":"https://api.github.com/repos/terraform-providers/terraform-provider-google/issues/1356/comments","events_url":"https://api.github.com/repos/terraform-providers/terraform-provider-google/issues/1356/events","html_url":"https://github.com/terraform-providers/terraform-provider-google/issues/1356","id":315820678,"node_id":"MDU6SXNzdWUzMTU4MjA2Nzg=","number":1356,"title":"google_project_iam_binding & google_project_service incorrect destruction order","user":{"login":"hawksight","id":11051419,"node_id":"MDQ6VXNlcjExMDUxNDE5","avatar_url":"https://avatars1.githubusercontent.com/u/11051419?v=4","gravatar_id":"","url":"https://api.github.com/users/hawksight","html_url":"https://github.com/hawksight","followers_url":"https://api.github.com/users/hawksight/followers","following_url":"https://api.github.com/users/hawksight/following{/other_user}","gists_url":"https://api.github.com/users/hawksight/gists{/gist_id}","starred_url":"https://api.github.com/users/hawksight/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hawksight/subscriptions","organizations_url":"https://api.github.com/users/hawksight/orgs","repos_url":"https://api.github.com/users/hawksight/repos","events_url":"https://api.github.com/users/hawksight/events{/privacy}","received_events_url":"https://api.github.com/users/hawksight/received_events","type":"User","site_admin":false},"labels":[{"id":620326519,"node_id":"MDU6TGFiZWw2MjAzMjY1MTk=","url":"https://api.github.com/repos/terraform-providers/terraform-provider-google/labels/bug","name":"bug","color":"f7c6c7","default":true},{"id":620326989,"node_id":"MDU6TGFiZWw2MjAzMjY5ODk=","url":"https://api.github.com/repos/terraform-providers/terraform-provider-google/labels/documentation","name":"documentation","color":"fef2c0","default":false},{"id":670571367,"node_id":"MDU6TGFiZWw2NzA1NzEzNjc=","url":"https://api.github.com/repos/terraform-providers/terraform-provider-google/labels/upstream-terraform","name":"upstream-terraform","color":"cccccc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-04-19T10:35:37Z","updated_at":"2018-07-06T11:44:10Z","closed_at":null,"author_association":"NONE","body":"### Terraform Version\r\n```\r\nTerraform v0.11.7\r\n+ provider.google v1.9.0\r\n+ provider.null v1.0.0\r\n```\r\n\r\n### Affected Resource(s)\r\n- google_project_iam_binding \r\n- google_project_service\r\n\r\n\r\n### Terraform Configuration Files\r\nThe following is an extract but gets the gist for the main issue here:\r\n```hcl\r\n# --- Resource Configuration --- #\r\nresource \"google_project\" \"project\" {\r\n  name            = \"some_project\"\r\n  project_id      = \"some_project\"\r\n  billing_account = \"${var.GCLOUD_BILLING_ID}\"\r\n  org_id          = \"${var.GCLOUD_ORG_ID}\"\r\n  skip_delete     = \"true\"\r\n}\r\n\r\n/* Add admins to the project from something email group\r\n */\r\nresource \"google_project_iam_binding\" \"admins\" {\r\n  project = \"${google_project.project.project_id}\"\r\n  role    = \"roles/owner\"\r\n\r\n  members = [\r\n    \"group:something@example.com\",\r\n    \"serviceAccount:terraform@some_other_project.iam.gserviceaccount.com\"\r\n  ]\r\n}\r\n\r\n/* Need to enable apis / services\r\n * Docs: https://www.terraform.io/docs/providers/google/r/google_project_service.html\r\n * This will enable extras and leave defaults enabled\r\n */\r\nresource \"google_project_service\" \"enable\" {\r\n  count   = \"${length(var.PROJECT_SERVICES)}\"\r\n  project = \"${google_project.project.project_id}\"\r\n  service = \"${element(var.PROJECT_SERVICES, count.index)}\"\r\n}\r\n\r\n/* This makes following errors because of the async nature of enabling\r\n * apis within gcloud. So ensues a hack!\r\n */\r\nresource \"null_resource\" \"wait\" {\r\n  # Use this to wait a minute\r\n  provisioner \"local-exec\" {\r\n    command     = \"import time; time.sleep(60))\"\r\n    interpreter = [\"python3\", \"-c\"]\r\n  }\r\n  depends_on = [\"google_project_service.enable\"]\r\n}\r\n\r\n/* Now make some actual resources to serve infrastructure\r\n */\r\nmodule \"network\" {...}\r\n\r\nmodule \"cluster\" {...}\r\n\r\nmodule \"lb-http\" {...}\r\n```\r\n\r\n### Debug Output\r\nThe plan to destroy: `terraform plan -var-file=inputs.tfvars -out plan -destroy`\r\n\r\n```\r\nRefreshing Terraform state in-memory prior to plan...\r\nThe refreshed state will be used to calculate this plan, but will not be\r\npersisted to local or remote state storage.\r\n\r\nnull_resource.np: Refreshing state... (ID: 2454169200766082532)\r\ngoogle_project.project: Refreshing state... (ID: some_project)\r\ndata.google_compute_lb_ip_ranges.lb: Refreshing state...\r\ngoogle_project_iam_binding.admins: Refreshing state... (ID: some_project/roles/owner)\r\ngoogle_project_service.enable[0]: Refreshing state... (ID: some_project/compute.googleapis.com)\r\ngoogle_compute_network.vpc: Refreshing state... (ID: vpc-ops)\r\ngoogle_project_service.enable[1]: Refreshing state... (ID: some_project/container.googleapis.com)\r\ngoogle_compute_subnetwork.subnet: Refreshing state... (ID: europe-west2/vpc-ops-subnet1)\r\ngoogle_compute_health_check.default: Refreshing state... (ID: vpc-ops-lb-backend-0)\r\ngoogle_compute_firewall.ingress: Refreshing state... (ID: vpc-ops-k8s-ingress)\r\ngoogle_compute_firewall.default-hc: Refreshing state... (ID: vpc-ops-lb-hc-0)\r\ngoogle_compute_global_address.default: Refreshing state... (ID: vpc-ops-lb-address)\r\ngoogle_compute_firewall.fw-lb: Refreshing state... (ID: vpc-ops-lb-allow-web)\r\ngoogle_container_cluster.cluster: Refreshing state... (ID: k8s-ops)\r\ngoogle_container_node_pool.pools: Refreshing state... (ID: europe-west2-a/k8s-ops/vpc-ops-np-1)\r\ngoogle_compute_backend_service.default: Refreshing state... (ID: vpc-ops-lb-backend-0)\r\ngoogle_compute_url_map.custom: Refreshing state... (ID: some_project-url-map)\r\ngoogle_compute_target_http_proxy.default: Refreshing state... (ID: vpc-ops-lb-http-proxy)\r\ngoogle_compute_global_forwarding_rule.http: Refreshing state... (ID: vpc-ops-lb-http)\r\n\r\n------------------------------------------------------------------------\r\n\r\nAn execution plan has been generated and is shown below.\r\nResource actions are indicated with the following symbols:\r\n  - destroy\r\n\r\nTerraform will perform the following actions:\r\n\r\n  - google_compute_url_map.custom\r\n\r\n  - google_project.project\r\n\r\n  - google_project_iam_binding.admins\r\n\r\n  - google_project_service.enable[0]\r\n\r\n  - google_project_service.enable[1]\r\n\r\n  - module.cluster.google_compute_firewall.ingress\r\n\r\n  - module.cluster.google_container_cluster.cluster\r\n\r\n  - module.cluster.google_container_node_pool.pools\r\n\r\n  - module.cluster.null_resource.np\r\n\r\n  - module.lb-http.google_compute_backend_service.default\r\n\r\n  - module.lb-http.google_compute_firewall.default-hc\r\n\r\n  - module.lb-http.google_compute_firewall.fw-lb\r\n\r\n  - module.lb-http.google_compute_global_address.default\r\n\r\n  - module.lb-http.google_compute_global_forwarding_rule.http\r\n\r\n  - module.lb-http.google_compute_health_check.default\r\n\r\n  - module.lb-http.google_compute_target_http_proxy.default\r\n\r\n  - module.network.google_compute_network.vpc\r\n\r\n  - module.network.google_compute_subnetwork.subnet\r\n\r\n\r\nPlan: 0 to add, 0 to change, 18 to destroy.\r\n\r\n------------------------------------------------------------------------\r\n\r\nThis plan was saved to: plan\r\n\r\nTo perform exactly these actions, run the following command to apply:\r\n    terraform apply \"plan\"\r\n```\r\n\r\nApply output\r\n```\r\ngoogle_project_iam_binding.admins: Destroying... (ID: some_project/roles/owner)\r\ngoogle_project_service.enable[1]: Destroying... (ID: some_project/container.googleapis.com)\r\ngoogle_project_service.enable[0]: Destroying... (ID: some_project/compute.googleapis.com)\r\nmodule.lb-http.google_compute_global_forwarding_rule.http: Destroying... (ID: vpc-ops-lb-http)\r\nmodule.cluster.google_compute_firewall.ingress: Destroying... (ID: vpc-ops-k8s-ingress)\r\nmodule.lb-http.google_compute_firewall.fw-lb: Destroying... (ID: vpc-ops-lb-allow-web)\r\nmodule.cluster.google_container_node_pool.pools: Destroying... (ID: europe-west2-a/k8s-ops/vpc-ops-np-1)\r\nmodule.lb-http.google_compute_firewall.default-hc: Destroying... (ID: vpc-ops-lb-hc-0)\r\ngoogle_project_iam_binding.admins: Destruction complete after 6s\r\nmodule.lb-http.google_compute_firewall.fw-lb: Still destroying... (ID: vpc-ops-lb-allow-web, 10s elapsed)\r\nmodule.cluster.google_container_node_pool.pools: Still destroying... (ID: europe-west2-a/k8s-ops/vpc-ops-np-1, 10s elapsed)\r\ngoogle_project_service.enable.1: Still destroying... (ID: some_project/container.googleapis.com, 10s elapsed)\r\nmodule.lb-http.google_compute_firewall.default-hc: Still destroying... (ID: vpc-ops-lb-hc-0, 10s elapsed)\r\nmodule.cluster.google_compute_firewall.ingress: Still destroying... (ID: vpc-ops-k8s-ingress, 10s elapsed)\r\nmodule.lb-http.google_compute_global_forwarding_rule.http: Still destroying... (ID: vpc-ops-lb-http, 10s elapsed)\r\ngoogle_project_service.enable.0: Still destroying... (ID: some_project/compute.googleapis.com, 10s elapsed)\r\nmodule.lb-http.google_compute_global_forwarding_rule.http: Destruction complete after 11s\r\nmodule.lb-http.google_compute_target_http_proxy.default: Destroying... (ID: vpc-ops-lb-http-proxy)\r\ngoogle_project_service.enable[1]: Destruction complete after 12s\r\nmodule.cluster.google_compute_firewall.ingress: Destruction complete after 13s\r\nmodule.lb-http.google_compute_firewall.default-hc: Destruction complete after 17s\r\nmodule.lb-http.google_compute_firewall.fw-lb: Destruction complete after 18s\r\nmodule.lb-http.google_compute_global_address.default: Destroying... (ID: vpc-ops-lb-address)\r\nmodule.cluster.google_container_node_pool.pools: Still destroying... (ID: europe-west2-a/k8s-ops/vpc-ops-np-1, 20s elapsed)\r\nmodule.lb-http.google_compute_target_http_proxy.default: Still destroying... (ID: vpc-ops-lb-http-proxy, 10s elapsed)\r\nmodule.lb-http.google_compute_target_http_proxy.default: Destruction complete after 12s\r\ngoogle_compute_url_map.custom: Destroying... (ID: some_project-url-map)\r\nmodule.lb-http.google_compute_global_address.default: Still destroying... (ID: vpc-ops-lb-address, 10s elapsed)\r\nmodule.lb-http.google_compute_global_address.default: Destruction complete after 10s\r\nmodule.cluster.google_container_node_pool.pools: Still destroying... (ID: europe-west2-a/k8s-ops/vpc-ops-np-1, 30s elapsed)\r\ngoogle_compute_url_map.custom: Still destroying... (ID: some_project-url-map, 10s elapsed)\r\ngoogle_compute_url_map.custom: Destruction complete after 11s\r\nmodule.lb-http.google_compute_backend_service.default: Destroying... (ID: vpc-ops-lb-backend-0)\r\nmodule.cluster.google_container_node_pool.pools: Still destroying... (ID: europe-west2-a/k8s-ops/vpc-ops-np-1, 40s elapsed)\r\nmodule.lb-http.google_compute_backend_service.default: Still destroying... (ID: vpc-ops-lb-backend-0, 10s elapsed)\r\nmodule.lb-http.google_compute_backend_service.default: Destruction complete after 11s\r\nmodule.lb-http.google_compute_health_check.default: Destroying... (ID: vpc-ops-lb-backend-0)\r\nmodule.cluster.google_container_node_pool.pools: Still destroying... (ID: europe-west2-a/k8s-ops/vpc-ops-np-1, 50s elapsed)\r\nmodule.lb-http.google_compute_health_check.default: Still destroying... (ID: vpc-ops-lb-backend-0, 10s elapsed)\r\nmodule.lb-http.google_compute_health_check.default: Destruction complete after 10s\r\nmodule.cluster.google_container_node_pool.pools: Still destroying... (ID: europe-west2-a/k8s-ops/vpc-ops-np-1, 1m0s elapsed)\r\n\r\nError: Error applying plan:\r\n\r\n3 error(s) occurred:\r\n\r\n* google_project_service.enable[0] (destroy): 1 error(s) occurred:\r\n\r\n* google_project_service.enable.0: Error disabling service: Error disabling service \"compute.googleapis.com\" for project \"some_project\": Error code 9, message: Could not turn off service, as it still has resources in use.:[compute.googleapis.com]\r\n* module.cluster.output.K8S_INSTANCE_GROUP_URLS: Resource 'data.google_container_cluster.information' does not have attribute 'instance_group_urls' for variable 'data.google_container_cluster.information.instance_group_urls'\r\n* module.cluster.google_container_node_pool.pools (destroy): 1 error(s) occurred:\r\n\r\n* google_container_node_pool.pools: Error waiting for deleting GKE NodePool: googleapi: Error 403: Required \"container.operations.get\" permission for \"projects/some_project/zones/europe-west2-a/operations/operation-1524131461021-7acd4173\"., forbidden\r\n```\r\n\r\n### Expected Behavior\r\nThe resources `google_project_iam_binding` and `google_project_service` should have been the very last resources to be deleted. Ie.\r\n* everything else\r\n* `google_project_service`\r\n* `google_project_iam_binding` - This has to be last because `google_project_iam_binding` will overwrite the 'owner' role when re-applying terraform. So if you try and add people using this, it will remove owners not provided in here. Hence all owners are assigned using a count with this feature. Including the service account that I use to create the project and everything else.\r\n\r\ntl;dr - `google_project_iam_binding` overwrites role membership rather than appending. I should have used `google_project_iam_member` instead\r\n\r\n### Actual Behavior\r\nTerraform seemed to go free range and just start deleting things. The two most important resources are deleted first! Which leads to subsequent issues of:\r\n1) Now being locked out of a project because its removed me and the service account.\r\n2) Terraform cannot remove other resources because its disabled services. It's also removed the service account performing the actions from the project, so even if the services were still enabled.. there's no longer an account to run with. You can see the error trying to remove a node pool.\r\n\r\n### Steps to Reproduce\r\n1. Terraform config that creates , project, iam_binding, enable 'compute' service, and goes and builds a compute instance in project created. Preferably a module for the compute instance bit as it will show up where any `depends_on` flaws + my infra is all based on various modules. (see my hcl example for reference)\r\n2. Build it out.\r\n3. Try and destroy everything - see what order terraform tries to do things\r\n\r\n### Important Factoids\r\n#### IAM\r\nI initially only used `google_project_iam_binding` to add owners (I did not use the service account in here). However after creation of the project and adding this permission, I tried to reapply and noticed that it was going to remove the service account, and add the new permissions instead.\r\nHence why in this scenario, I added the service account into this object (using count) as well as the additional members.\r\n\r\nI used `google_project_iam_binding` rather than `google_project_iam_policy` explicitly because I noted this warning:\r\n![screen shot 2018-04-19 at 11 16 23](https://user-images.githubusercontent.com/11051419/38986084-453a454a-43c3-11e8-83e3-216a8c0cbfc5.png)\r\n\r\nObviously it seems you can also lock yourself out with the `google_project_iam_binding` resource if not used correctly (I'm guessing there are some simple tricks to avoid what I have done). Perhaps the warning should be for all resources as I assumed it was specific to `policy`.\r\n\r\n#### Outputs\r\nThere's also an error there around a data object:\r\n```\r\n* module.cluster.output.K8S_INSTANCE_GROUP_URLS: Resource 'data.google_container_cluster.information' does not have attribute 'instance_group_urls' for variable 'data.google_container_cluster.information.instance_group_urls'\r\n```\r\nPerhaps this might indicate an ordering issue or issue with data objects. In my case the data object is used post nope_pool creation. This is to trap all instance_group_urls.\r\n```hcl\r\ndata \"google_container_cluster\" \"information\" {\r\n  name       = \"${google_container_cluster.cluster.name}\"\r\n  zone       = \"${google_container_cluster.cluster.zone}\"\r\n  project    = \"${var.META_PROJECT}\"\r\n  depends_on = [\"google_container_node_pool.pools\"]\r\n}\r\n\r\noutput \"K8S_INSTANCE_GROUP_URLS\" {\r\n  value       = \"${data.google_container_cluster.information.instance_group_urls}\"\r\n  description = \"URLs to the instance groups for all nodes\"\r\n}\r\n```\r\n\r\nI actually don't care about this during destruction. I can't think of a scenario where I would be. I've had this issue a little while and usually something like `export TF_WARN_OUTPUT_ERRORS=1` will ignore this - but the default behaviour in this scenario I think should be just to ignore the output entirely. Im destroying it, which should mean 'I accept there's not going to be an output because there will be no data when the cluster is destroyed'.\r\n\r\nIt looks like its trying to get data on something its destroyed in in the process of destroying. Perhaps the simple fix is to add a flag to not lookup data object on destroy?\r\n\r\n#### My Modules\r\nMy modules all work when used inside a project and giving the code for that means much more copy paste. And it doesn't add anything particular to this issue.\r\n\r\n### References\r\n#1292 - Slightly related in the discussion about how dependency on deletion is formed. `service` and `iam_member` resources both seem to get deleted early on."},"comment":{"url":"https://api.github.com/repos/terraform-providers/terraform-provider-google/issues/comments/403010311","html_url":"https://github.com/terraform-providers/terraform-provider-google/issues/1356#issuecomment-403010311","issue_url":"https://api.github.com/repos/terraform-providers/terraform-provider-google/issues/1356","id":403010311,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMzAxMDMxMQ==","user":{"login":"paddycarver","id":59249,"node_id":"MDQ6VXNlcjU5MjQ5","avatar_url":"https://avatars0.githubusercontent.com/u/59249?v=4","gravatar_id":"","url":"https://api.github.com/users/paddycarver","html_url":"https://github.com/paddycarver","followers_url":"https://api.github.com/users/paddycarver/followers","following_url":"https://api.github.com/users/paddycarver/following{/other_user}","gists_url":"https://api.github.com/users/paddycarver/gists{/gist_id}","starred_url":"https://api.github.com/users/paddycarver/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/paddycarver/subscriptions","organizations_url":"https://api.github.com/users/paddycarver/orgs","repos_url":"https://api.github.com/users/paddycarver/repos","events_url":"https://api.github.com/users/paddycarver/events{/privacy}","received_events_url":"https://api.github.com/users/paddycarver/received_events","type":"User","site_admin":false},"created_at":"2018-07-06T11:44:10Z","updated_at":"2018-07-06T11:44:10Z","author_association":"CONTRIBUTOR","body":"Our IAM docs have been updated since this was opened, but I tried to apply the spirit of it to #1735. I opted not to call out the dependency ordering info, as that's already covered in [the getting started guide for Terraform](https://www.terraform.io/intro/getting-started/dependencies.html#implicit-and-explicit-dependencies), so I think it's inappropriate to document it at the provider level. I also worry about losing the forest for the trees in our documentation by being too verbose; after all, a warning to be careful about order of deletion could apply to most our resources, which could return errors if they're not deleted in the proper order. I think that information is better suited in the getting started guide, and that the provider focus on documenting what's unique to the provider."}}
{"actor":{"display_login":"paddycarver"}}