{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/terraform/issues/2957","repository_url":"https://api.github.com/repos/hashicorp/terraform","labels_url":"https://api.github.com/repos/hashicorp/terraform/issues/2957/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/terraform/issues/2957/comments","events_url":"https://api.github.com/repos/hashicorp/terraform/issues/2957/events","html_url":"https://github.com/hashicorp/terraform/issues/2957","id":99557259,"number":2957,"title":"Error when instance changed that has EBS volume attached","user":{"login":"bloopletech","id":13609,"avatar_url":"https://avatars.githubusercontent.com/u/13609?v=3","gravatar_id":"","url":"https://api.github.com/users/bloopletech","html_url":"https://github.com/bloopletech","followers_url":"https://api.github.com/users/bloopletech/followers","following_url":"https://api.github.com/users/bloopletech/following{/other_user}","gists_url":"https://api.github.com/users/bloopletech/gists{/gist_id}","starred_url":"https://api.github.com/users/bloopletech/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bloopletech/subscriptions","organizations_url":"https://api.github.com/users/bloopletech/orgs","repos_url":"https://api.github.com/users/bloopletech/repos","events_url":"https://api.github.com/users/bloopletech/events{/privacy}","received_events_url":"https://api.github.com/users/bloopletech/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hashicorp/terraform/labels/bug","name":"bug","color":"f7c6c7"},{"url":"https://api.github.com/repos/hashicorp/terraform/labels/provider/aws","name":"provider/aws","color":"bfdadc"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":48,"created_at":"2015-08-07T01:32:27Z","updated_at":"2016-08-10T00:00:03Z","closed_at":null,"body":"This is the specific error I get from terraform:\r\n\r\n````\r\naws_volume_attachment.admin_rundeck: Destroying...\r\naws_volume_attachment.admin_rundeck: Error: 1 error(s) occurred:\r\n\r\n* Error waiting for Volume (<vol id>) to detach from Instance: <instance id>\r\nError applying plan:\r\n\r\n3 error(s) occurred:\r\n\r\n* Error waiting for Volume (<vol id>) to detach from Instance: <instance id>\r\n* aws_instance.admin_rundeck: diffs didn't match during apply. This is a bug with Terraform and should be reported.\r\n* aws_volume_attachment.admin_rundeck: diffs didn't match during apply. This is a bug with Terraform and should be reported.\r\n\r\nTerraform does not automatically rollback in the face of errors.\r\nInstead, your Terraform state file has been partially updated with\r\nany resources that successfully completed. Please address the error\r\nabove and apply again to incrementally change your infrastructure.\r\n````\r\n\r\nWe are building out some infrastructure in EC2 using terraform (v0.6.0). I'm currently working out our persistent storage setup. The strategy I'm planning is to have the root volume of every instance be ephemeral, and to move all persistent data to a separate EBS volume (one persistent volume per instance). We want this to be as automated as possible of course.\r\n\r\nHere is a relevant excerpt from our terraform config:\r\n\r\n````\r\nresource \"aws_instance\" \"admin_rundeck\" {\r\n  ami = \"${var.aws_ami_rundeck}\"\r\n  instance_type = \"${var.aws_instance_type}\"\r\n  subnet_id = \"${aws_subnet.admin_private.id}\"\r\n  vpc_security_group_ids = [\"${aws_security_group.base.id}\", \"${aws_security_group.admin_rundeck.id}\"]\r\n  key_name = \"Administration\"\r\n\r\n  root_block_device {\r\n    delete_on_termination = false\r\n  }\r\n\r\n  tags {\r\n    Name = \"admin-rundeck-01\"\r\n    Role = \"rundeck\"\r\n    Application = \"rundeck\"\r\n    Project = \"Administration\"\r\n  }\r\n}\r\n\r\nresource \"aws_ebs_volume\" \"admin_rundeck\" {\r\n  size = 500\r\n  availability_zone = \"${var.default_aws_az}\"\r\n  snapshot_id = \"snap-66fc2258\"\r\n  tags = {\r\n    Name = \"Rundeck Data Volume\"\r\n  }\r\n}\r\n\r\nresource \"aws_volume_attachment\" \"admin_rundeck\" {\r\n  device_name = \"/dev/xvdf\"\r\n  instance_id = \"${aws_instance.admin_rundeck.id}\"\r\n  volume_id = \"${aws_ebs_volume.admin_rundeck.id}\"\r\n\r\n  depends_on = \"aws_route53_record.admin_rundeck\"\r\n\r\n  connection {\r\n    host = \"admin-rundeck-01.<domain name>\"\r\n    bastion_host = \"${aws_instance.admin_jumpbox.public_ip}\"\r\n    timeout = \"1m\"\r\n    key_file = \"~/.ssh/admin.pem\"\r\n    user = \"ubuntu\"\r\n  }\r\n\r\n  provisioner \"remote-exec\" {\r\n    script = \"mount.sh\"\r\n  }\r\n\r\n  provisioner \"remote-exec\" {\r\n    inline = [\r\n      \"sudo mkdir -m 2775 /data/rundeck\",\r\n      \"sudo mkdir /data/rundeck/data /data/rundeck/projects && sudo chown -R rundeck:rundeck /data/rundeck\",\r\n      \"sudo service rundeckd restart\"\r\n    ]\r\n  }\r\n}\r\n````\r\n\r\nAnd mount.sh:\r\n\r\n````\r\n#!/bin/bash\r\n\r\nwhile [ ! -e /dev/xvdf ]; do sleep 1; done\r\n\r\nfstab_string='/dev/xvdf /data ext4 defaults,nofail,nobootwait 0 2'\r\nif grep -q -F -v \"$fstab_string\" /etc/fstab; then\r\n  echo \"$fstab_string\" | sudo tee -a /etc/fstab\r\nfi\r\n\r\nsudo mkdir -p /data && sudo mount -t ext4 /dev/xvdf /data\r\n````\r\n\r\nAs you can see, this:\r\n\r\n* Provisions an instance to run Rundeck (http://rundeck.org/)\r\n* Provisions an EBS volume based off of a snapshot. The snapshot in this case is just an empty ext4 partition.\r\n* Attaches the voulme to the instance\r\n* Mounts the volume inside the instance, and then creates some directories to store data in\r\n\r\nThis works fine the first time it's run. But any time we:\r\n\r\n* make a change to the instance configuration (i.e. change the value of var.aws_ami_rundeck) or\r\n* make a change to the provisioner config of the volume attachment resource\r\n\r\nTerraform then tries to detach the extant volume from the instance, and this task fails every time. I believe this is because you are meant to unmount the ebs volume from inside the instance before detaching the volume. The problem is, I can't work out how to get terraform to unmount the volume inside the instance *before* trying to detach the volume.\r\n\r\nIt's almost like I need a provisioner to run before the resource is created, or a provisioner to run on destroy (obviously https://github.com/hashicorp/terraform/issues/386 comes to mind).\r\n\r\nThis feels like it would be a common problem for anyone working with persistent EBS volumes using terraform, but my googling hasn't really found anyone even having this problem.\r\n\r\nAm I simply doing it wrong? I'm not worried about how I get there specifically, I just would like to be able to provision persistent EBS volumes, and then attach and detach that volume to my instances in an automated fashion."},"comment":{"url":"https://api.github.com/repos/hashicorp/terraform/issues/comments/238728868","html_url":"https://github.com/hashicorp/terraform/issues/2957#issuecomment-238728868","issue_url":"https://api.github.com/repos/hashicorp/terraform/issues/2957","id":238728868,"user":{"login":"luckymike","id":479368,"avatar_url":"https://avatars.githubusercontent.com/u/479368?v=3","gravatar_id":"","url":"https://api.github.com/users/luckymike","html_url":"https://github.com/luckymike","followers_url":"https://api.github.com/users/luckymike/followers","following_url":"https://api.github.com/users/luckymike/following{/other_user}","gists_url":"https://api.github.com/users/luckymike/gists{/gist_id}","starred_url":"https://api.github.com/users/luckymike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/luckymike/subscriptions","organizations_url":"https://api.github.com/users/luckymike/orgs","repos_url":"https://api.github.com/users/luckymike/repos","events_url":"https://api.github.com/users/luckymike/events{/privacy}","received_events_url":"https://api.github.com/users/luckymike/received_events","type":"User","site_admin":false},"created_at":"2016-08-10T00:00:03Z","updated_at":"2016-08-10T00:00:03Z","body":"I agree with @LeslieCarr. \r\n\r\nWhile I personally think that long-lived EBS volumes that move between instances are a painful, compromise solution that rarely address the root issues of data-persistence, to the extent that there are many scenarios where people choose to do this, I absolutely expect my provisioner to be capable of managing those."}}
{"actor":{"display_login":"luckymike"}}