{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/terraform/issues/5335","repository_url":"https://api.github.com/repos/hashicorp/terraform","labels_url":"https://api.github.com/repos/hashicorp/terraform/issues/5335/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/terraform/issues/5335/comments","events_url":"https://api.github.com/repos/hashicorp/terraform/issues/5335/events","html_url":"https://github.com/hashicorp/terraform/issues/5335","id":136563646,"number":5335,"title":"Intermittent AWS Eventual Consistency Issues on CircleCI?","user":{"login":"josh-padnick","id":4295964,"avatar_url":"https://avatars.githubusercontent.com/u/4295964?v=3","gravatar_id":"","url":"https://api.github.com/users/josh-padnick","html_url":"https://github.com/josh-padnick","followers_url":"https://api.github.com/users/josh-padnick/followers","following_url":"https://api.github.com/users/josh-padnick/following{/other_user}","gists_url":"https://api.github.com/users/josh-padnick/gists{/gist_id}","starred_url":"https://api.github.com/users/josh-padnick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josh-padnick/subscriptions","organizations_url":"https://api.github.com/users/josh-padnick/orgs","repos_url":"https://api.github.com/users/josh-padnick/repos","events_url":"https://api.github.com/users/josh-padnick/events{/privacy}","received_events_url":"https://api.github.com/users/josh-padnick/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hashicorp/terraform/labels/bug","name":"bug","color":"f7c6c7"},{"url":"https://api.github.com/repos/hashicorp/terraform/labels/provider/aws","name":"provider/aws","color":"bfdadc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-02-26T01:08:10Z","updated_at":"2016-05-31T03:56:11Z","closed_at":null,"body":"I have a terraform configuration whose job is to create a VPC.  The basic structure is this:\r\n\r\n```\r\nprod-vpc\r\n- creates vpc\r\n- adds vpc peering route with mgmt-vpc\r\n\r\nstage-vpc\r\n- creates vpc\r\n- adds vpc peering route with mgmt-vpc\r\n\r\nmgmt-vpc\r\n- adds vpc peering routes with prod and stage\r\n\r\ncreate-vpc-peering-connections\r\n```\r\n\r\nAs you can see, there are many interdependencies, however `apply`ing and `destroy`ing this from my macbook consistently works fine.\r\n\r\nWhen I run this same terraform configuration as part of a CircleCI build, however, I get intermittent failures.  Here are some of the errors I've gotten:\r\n\r\n```\r\n* aws_route.internet-gateway: \r\nerror finding matching route for Route table (rtb-dff811b8) and destination CIDR block (0.0.0.0/0)\r\n```\r\n\r\n```\r\n* aws_route.vpc-peering.1: \r\nerror finding matching route for Route table (rtb-a39b72c4) and destination CIDR block (10.110.0.0/18)\r\n```\r\n\r\nNotice that sometimes I'm getting the same error, but for a different resource, though the second one occurred twice.  Here are some other errors.  Note that each build will fail with one of these, or in some rare cases, succeed.\r\n\r\n\r\n\r\n```\r\n4 error(s) occurred:\r\n\r\n* aws_internet_gateway.main: InvalidInternetGatewayID.NotFound: The internetGateway ID 'igw-eab0118e' does not exist\r\n\tstatus code: 400, request id: \r\n* Resource 'aws_eip.nat' does not have attribute 'id' for variable 'aws_eip.nat.id'\r\n* aws_route.vpc-peering.0: \r\nerror finding matching route for Route table (rtb-eb99708c) and destination CIDR block (10.100.0.0/18)\r\n* aws_route.vpc-peering.1: \r\nerror finding matching route for Route table (rtb-eb99708c) and destination CIDR block (10.110.0.0/18)\r\n```\r\n\r\n```\r\n1 error(s) occurred:\r\n\r\n* aws_nat_gateway.nat.0: Error waiting for NAT Gateway (nat-0526389328c14da4e) to become available: unexpected state 'failed', wanted target '[available]'\r\n```\r\n\r\n```\r\n4 error(s) occurred:\r\n\r\n* aws_route.internet-gateway: \r\nerror finding matching route for Route table (rtb-5ca64f3b) and destination CIDR block (0.0.0.0/0)\r\n* Resource 'aws_eip.nat' does not have attribute 'id' for variable 'aws_eip.nat.*.id'\r\n* Resource 'aws_eip.nat' does not have attribute 'id' for variable 'aws_eip.nat.*.id'\r\n* Resource 'aws_eip.nat' does not have attribute 'id' for variable 'aws_eip.nat.*.id'\r\n```\r\n\r\n\r\n\r\nEvery so often the build succeeds, but even then I sometimes receive some non-fatal `diffs didn't match during apply` warnings:\r\n\r\n```\r\n* aws_iam_instance_profile.instance_profile: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.\r\n\r\nPlease include the following information in your report:\r\n\r\nTerraform Version: 0.6.12\r\n    Resource ID: aws_iam_instance_profile.instance_profile\r\n    Mismatch reason: attribute mismatch: roles.2877750799\r\n    Diff One (usually from plan): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{\"create_date\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, \"unique_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, \"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"lc-instance-profile-${var.app_name}-${var.vpc_name}\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Type:0x0}, \"path\":*terraform.ResourceAttrDiff{Old:\"\", New:\"/\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Type:0x0}, \"roles.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, \"roles.2877750799\":*terraform.ResourceAttrDiff{Old:\"\", New:\"lc-instance-role-${var.app_name}-${var.vpc_name}\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, \"arn\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}\r\n    Diff Two (usually from apply): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{\"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"lc-instance-profile-asg-example-app-stg\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Type:0x0}, \"path\":*terraform.ResourceAttrDiff{Old:\"\", New:\"/\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Type:0x0}, \"roles.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, \"roles.4293975883\":*terraform.ResourceAttrDiff{Old:\"\", New:\"lc-instance-role-asg-example-app-stg\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, \"arn\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, \"create_date\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, \"unique_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}\r\n```\r\n\r\n\r\n\r\nSo, based on these data points, and the fact that it runs fine in my localdev, I'm guessing there's something about the CircleCI environment.  Most (all?) of these errors indicate AWS eventual consistency issues, but I'm struggling to explain why the CircleCI environment would be more likely to trigger them?\r\n\r\nSome other datapoints:\r\n\r\n- My localdev is a Macbook Pro with 8 cores\r\n- My localdev's `GOMAXPROCS` = 27984\r\n- CircleCI's environment has 32 cores (found from SSHinto into the build)\r\n- CircleCI's `GOMAXPROCS` = ~4000000\r\n- CircleCI's environment is running Ubuntu (I believe 14.04 LTS)\r\n- Setting `GOMAXPROCS` = 1 via an env var in CircleCI did yield a successful build, but obviously slowed things down.\r\n- Using `terraform apply -parallelism=3` (just to reduce this down from 10) did not seem to affect things, but I'm guessing `GOMAXPROCS` and `parallelism` ultimately have the same effect.\r\n\r\nDoes anyone have any ideas for how I might further debug this or why this is happening?  Thanks for your help and for this outstanding piece of software!"},"comment":{"url":"https://api.github.com/repos/hashicorp/terraform/issues/comments/222587231","html_url":"https://github.com/hashicorp/terraform/issues/5335#issuecomment-222587231","issue_url":"https://api.github.com/repos/hashicorp/terraform/issues/5335","id":222587231,"user":{"login":"josh-padnick","id":4295964,"avatar_url":"https://avatars.githubusercontent.com/u/4295964?v=3","gravatar_id":"","url":"https://api.github.com/users/josh-padnick","html_url":"https://github.com/josh-padnick","followers_url":"https://api.github.com/users/josh-padnick/followers","following_url":"https://api.github.com/users/josh-padnick/following{/other_user}","gists_url":"https://api.github.com/users/josh-padnick/gists{/gist_id}","starred_url":"https://api.github.com/users/josh-padnick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josh-padnick/subscriptions","organizations_url":"https://api.github.com/users/josh-padnick/orgs","repos_url":"https://api.github.com/users/josh-padnick/repos","events_url":"https://api.github.com/users/josh-padnick/events{/privacy}","received_events_url":"https://api.github.com/users/josh-padnick/received_events","type":"User","site_admin":false},"created_at":"2016-05-31T03:56:11Z","updated_at":"2016-05-31T03:56:11Z","body":"@carlossg Thanks for submitting this! We have an automated test suite for our terraform templates and, as you indicate in #6775, the problem seems to have gotten worse lately, probably due to AWS's own issues handling load. Very excited about this getting merged."}}
{"actor":{"display_login":"josh-padnick"}}