{"action":"created","issue":{"url":"https://api.github.com/repos/docker/for-aws/issues/157","repository_url":"https://api.github.com/repos/docker/for-aws","labels_url":"https://api.github.com/repos/docker/for-aws/issues/157/labels{/name}","comments_url":"https://api.github.com/repos/docker/for-aws/issues/157/comments","events_url":"https://api.github.com/repos/docker/for-aws/issues/157/events","html_url":"https://github.com/docker/for-aws/issues/157","id":325780923,"node_id":"MDU6SXNzdWUzMjU3ODA5MjM=","number":157,"title":"Service fails to start with Cloudstor EBS Volume attached","user":{"login":"ambrons","id":1403700,"node_id":"MDQ6VXNlcjE0MDM3MDA=","avatar_url":"https://avatars0.githubusercontent.com/u/1403700?v=4","gravatar_id":"","url":"https://api.github.com/users/ambrons","html_url":"https://github.com/ambrons","followers_url":"https://api.github.com/users/ambrons/followers","following_url":"https://api.github.com/users/ambrons/following{/other_user}","gists_url":"https://api.github.com/users/ambrons/gists{/gist_id}","starred_url":"https://api.github.com/users/ambrons/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ambrons/subscriptions","organizations_url":"https://api.github.com/users/ambrons/orgs","repos_url":"https://api.github.com/users/ambrons/repos","events_url":"https://api.github.com/users/ambrons/events{/privacy}","received_events_url":"https://api.github.com/users/ambrons/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-23T16:22:50Z","updated_at":"2018-06-03T20:49:07Z","closed_at":null,"author_association":"NONE","body":"### Expected behavior\r\nService starts with attached EBS Volume attached\r\n### Actual behavior\r\n\r\nMy assumption is that it's taking too long to snapshot and load the EBS volume for a specific availability zone and therefore times out.\r\n\r\nNote: the EBS volumes are 200GB, however they're currently empty.\r\n\r\nThe initial error is this:\r\n```\r\n$ swarm service ps nvkmbnmc9nwyn8ojw3v1bkjzh --no-trunc\r\nID                          NAME                IMAGE                                                                                                                                                     NODE                                               DESIRED STATE       CURRENT STATE              ERROR                                                                                                                                                                       PORTS\r\nzgkin90vm4iyt242jsjeb80h4   cassandra.1         xxx.dkr.ecr.us-east-1.amazonaws.com/zco/esports/cassandra-swarm:latest@sha256:a7150a38203c44e332d05a37d8275a76a001be5b814c661d05fd73edab893437   ip-172-20-25-211.ap-southeast-1.compute.internal   Running             Preparing 13 seconds ago   \"Post http://%2Frun%2Fdocker%2Fplugins%2F4125eb31d4a89cab3863d96d60403c0134f69a0d19937225a2f0839f737384e1%2Fcloudstor.sock/VolumeDriver.Mount: context deadline exceeded\"   \r\n```\r\n\r\nAfter subsequent retires to start the service I get this error:\r\n```\r\n$ swarm service ps nvkmbnmc9nwyn8ojw3v1bkjzh --no-trunc\r\nID                          NAME                IMAGE                                                                                                                                                     NODE                                               DESIRED STATE       CURRENT STATE              ERROR                                                                                                                                                                                                                                                                                                                                                                                                                   PORTS\r\nep598hubcgsmuox2wm1wfcn1f   cassandra.1         xxx.dkr.ecr.us-east-1.amazonaws.com/zco/esports/cassandra-swarm:latest@sha256:a7150a38203c44e332d05a37d8275a76a001be5b814c661d05fd73edab893437   ip-172-20-9-77.ap-southeast-1.compute.internal     Running             Preparing 51 seconds ago                                                                                                                                                                                                                                                                                                                                                                                                                           \r\nzgkin90vm4iyt242jsjeb80h4    \\_ cassandra.1     xxx.dkr.ecr.us-east-1.amazonaws.com/zco/esports/cassandra-swarm:latest@sha256:a7150a38203c44e332d05a37d8275a76a001be5b814c661d05fd73edab893437   ip-172-20-25-211.ap-southeast-1.compute.internal   Shutdown            Rejected 51 seconds ago    \"create cassandra-1: found reference to volume 'cassandra-1' in driver 'cloudstor:aws', but got an error while checking the driver: error while checking if volume \"cassandra-1\" exists in driver \"cloudstor:aws\": Post http://%2Frun%2Fdocker%2Fplugins%2F4125eb31d4a89cab3863d96d60403c0134f69a0d19937225a2f0839f737384e1%2Fcloudstor.sock/VolumeDriver.Get: context deadline exceeded: volume name must be unique\"   \r\n```\r\n\r\nThe service never seems to start.\r\n\r\n### Information\r\n\r\nDocker-diagnose: 1527092193-JGugtUgVNBmvU7S8tXn0mV4ryIhPF4zc\r\n\r\nVolumes created:\r\n```\r\nswarm volume create -d \"cloudstor:aws\" --opt ebstype=io1 --opt size=200 --opt iops=1000 --opt backing=relocatable --opt ebs_tag_Name=cassandra-1 cassandra-1\r\n```\r\n\r\nAWS Region: ap-southeast-1\r\n\r\nService Creation Setup:\r\n```\r\ndocker service create \\\r\n  --name cassandra \\\r\n  --network data \\\r\n  --update-delay 60s \\\r\n  --replicas 1 \\\r\n  --with-registry-auth \\\r\n  --env LOCAL_JMX=no \\\r\n  --env SERVICE_NAME=cassandra \\\r\n  --constraint 'node.role != manager' \\\r\n  --reserve-memory 3gb \\\r\n  --mount type=volume,target=/var/lib/cassandra,source={{.Service.Name}}-{{.Task.Slot}} \\\r\n  xxx.dkr.ecr.us-east-1.amazonaws.com/zco/esports/cassandra-swarm\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/docker/for-aws/issues/comments/394190299","html_url":"https://github.com/docker/for-aws/issues/157#issuecomment-394190299","issue_url":"https://api.github.com/repos/docker/for-aws/issues/157","id":394190299,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NDE5MDI5OQ==","user":{"login":"soar","id":30570,"node_id":"MDQ6VXNlcjMwNTcw","avatar_url":"https://avatars0.githubusercontent.com/u/30570?v=4","gravatar_id":"","url":"https://api.github.com/users/soar","html_url":"https://github.com/soar","followers_url":"https://api.github.com/users/soar/followers","following_url":"https://api.github.com/users/soar/following{/other_user}","gists_url":"https://api.github.com/users/soar/gists{/gist_id}","starred_url":"https://api.github.com/users/soar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soar/subscriptions","organizations_url":"https://api.github.com/users/soar/orgs","repos_url":"https://api.github.com/users/soar/repos","events_url":"https://api.github.com/users/soar/events{/privacy}","received_events_url":"https://api.github.com/users/soar/received_events","type":"User","site_admin":false},"created_at":"2018-06-03T20:49:07Z","updated_at":"2018-06-03T20:49:07Z","author_association":"NONE","body":"Same problem - `cloudstor:aws` creates and deletes volumes successfully, but hangs when I try to start container.\r\n\r\n```\r\n# docker volume create -d \"cloudstor:aws\" --opt ebstype=gp2 --opt size=10 mylocalvol1\r\nmylocalvol1\r\n# docker volume ls\r\nDRIVER              VOLUME NAME\r\ncloudstor:aws       mylocalvol1\r\n# docker run -it -v mylocalvol1:/mnt debian bash\r\n... nothing after 10 minutes ...\r\n```"}}
{"actor":{"display_login":"soar"}}