{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/12106","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/12106/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/12106/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/12106/events","html_url":"https://github.com/rancher/rancher/issues/12106","id":305101582,"node_id":"MDU6SXNzdWUzMDUxMDE1ODI=","number":12106,"title":"EBS-Driver fails to mount AWS volume","user":{"login":"dadarakt","id":5033341,"node_id":"MDQ6VXNlcjUwMzMzNDE=","avatar_url":"https://avatars0.githubusercontent.com/u/5033341?v=4","gravatar_id":"","url":"https://api.github.com/users/dadarakt","html_url":"https://github.com/dadarakt","followers_url":"https://api.github.com/users/dadarakt/followers","following_url":"https://api.github.com/users/dadarakt/following{/other_user}","gists_url":"https://api.github.com/users/dadarakt/gists{/gist_id}","starred_url":"https://api.github.com/users/dadarakt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dadarakt/subscriptions","organizations_url":"https://api.github.com/users/dadarakt/orgs","repos_url":"https://api.github.com/users/dadarakt/repos","events_url":"https://api.github.com/users/dadarakt/events{/privacy}","received_events_url":"https://api.github.com/users/dadarakt/received_events","type":"User","site_admin":false},"labels":[{"id":294075522,"node_id":"MDU6TGFiZWwyOTQwNzU1MjI=","url":"https://api.github.com/repos/rancher/rancher/labels/area/storage","name":"area/storage","color":"fbca04","default":false},{"id":189155298,"node_id":"MDU6TGFiZWwxODkxNTUyOTg=","url":"https://api.github.com/repos/rancher/rancher/labels/status/resolved","name":"status/resolved","color":"0052cc","default":false},{"id":203009589,"node_id":"MDU6TGFiZWwyMDMwMDk1ODk=","url":"https://api.github.com/repos/rancher/rancher/labels/status/to-test","name":"status/to-test","color":"66aa66","default":false},{"id":618069126,"node_id":"MDU6TGFiZWw2MTgwNjkxMjY=","url":"https://api.github.com/repos/rancher/rancher/labels/version/1.6","name":"version/1.6","color":"e99695","default":false}],"state":"open","locked":false,"assignee":{"login":"soumyalj","id":18536626,"node_id":"MDQ6VXNlcjE4NTM2NjI2","avatar_url":"https://avatars1.githubusercontent.com/u/18536626?v=4","gravatar_id":"","url":"https://api.github.com/users/soumyalj","html_url":"https://github.com/soumyalj","followers_url":"https://api.github.com/users/soumyalj/followers","following_url":"https://api.github.com/users/soumyalj/following{/other_user}","gists_url":"https://api.github.com/users/soumyalj/gists{/gist_id}","starred_url":"https://api.github.com/users/soumyalj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumyalj/subscriptions","organizations_url":"https://api.github.com/users/soumyalj/orgs","repos_url":"https://api.github.com/users/soumyalj/repos","events_url":"https://api.github.com/users/soumyalj/events{/privacy}","received_events_url":"https://api.github.com/users/soumyalj/received_events","type":"User","site_admin":false},"assignees":[{"login":"soumyalj","id":18536626,"node_id":"MDQ6VXNlcjE4NTM2NjI2","avatar_url":"https://avatars1.githubusercontent.com/u/18536626?v=4","gravatar_id":"","url":"https://api.github.com/users/soumyalj","html_url":"https://github.com/soumyalj","followers_url":"https://api.github.com/users/soumyalj/followers","following_url":"https://api.github.com/users/soumyalj/following{/other_user}","gists_url":"https://api.github.com/users/soumyalj/gists{/gist_id}","starred_url":"https://api.github.com/users/soumyalj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumyalj/subscriptions","organizations_url":"https://api.github.com/users/soumyalj/orgs","repos_url":"https://api.github.com/users/soumyalj/repos","events_url":"https://api.github.com/users/soumyalj/events{/privacy}","received_events_url":"https://api.github.com/users/soumyalj/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/rancher/rancher/milestones/120","html_url":"https://github.com/rancher/rancher/milestone/120","labels_url":"https://api.github.com/repos/rancher/rancher/milestones/120/labels","id":3199200,"node_id":"MDk6TWlsZXN0b25lMzE5OTIwMA==","number":120,"title":"v1.6 - Jul 2018","description":"","creator":{"login":"deniseschannon","id":11410997,"node_id":"MDQ6VXNlcjExNDEwOTk3","avatar_url":"https://avatars1.githubusercontent.com/u/11410997?v=4","gravatar_id":"","url":"https://api.github.com/users/deniseschannon","html_url":"https://github.com/deniseschannon","followers_url":"https://api.github.com/users/deniseschannon/followers","following_url":"https://api.github.com/users/deniseschannon/following{/other_user}","gists_url":"https://api.github.com/users/deniseschannon/gists{/gist_id}","starred_url":"https://api.github.com/users/deniseschannon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deniseschannon/subscriptions","organizations_url":"https://api.github.com/users/deniseschannon/orgs","repos_url":"https://api.github.com/users/deniseschannon/repos","events_url":"https://api.github.com/users/deniseschannon/events{/privacy}","received_events_url":"https://api.github.com/users/deniseschannon/received_events","type":"User","site_admin":false},"open_issues":11,"closed_issues":20,"state":"open","created_at":"2018-03-19T16:51:28Z","updated_at":"2018-07-16T22:26:46Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2018-03-14T10:30:14Z","updated_at":"2018-07-16T23:02:13Z","closed_at":null,"author_association":"NONE","body":"**Rancher versions:**\r\nrancher/server: 1.6.14\r\nrancher/agent: 1.2.9\r\n\r\n**Infrastructure Stack versions:**\r\nhealthcheck: 0.3.3\r\nipsec: 0.13.7\r\nnetwork-services: 0.9.5\r\nscheduler: 0.8.3\r\nebs-driver: 0.8.11\r\n\r\n**Docker version**:  Server Version: 17.09.1-ce\r\n\r\n**Operating system and kernel:** 4.9.77-31.58.amzn1.x86_64\r\n\r\n**Type/provider of hosts:** AWS, hvm-hosts\r\n\r\n**Setup details:** single node rancher, internal DB\r\n\r\n**Environment Template:** Cattle 0.183.37\r\n\r\n**Steps to Reproduce:**\r\nAfter setting up the EBS driver for a stack, it is possible to create volumes for stacks in docker-compose and use it, i.e.:\r\n```yml\r\nvolumes:\r\n  test-data:\r\n    volumeType: gp2\r\n    size: 10\r\n```\r\nSnapshots created from this image could also be used to create new volumes like so:\r\n```yml\r\nvolumes:\r\n  test-data:\r\n    ec2_az: <availability-zone>\r\n    snapshotID: <snapshot-ID>\r\n    size: 10\r\n```\r\nWhen creating a snapshot off an existing EC2 volume used outside of Rancher or when creating a new volume in the AWS EC2 console in the right availability zone with the exact same parameters as above and then trying to use it in Rancher leads to the problem.\r\n```yml\r\nvolumes:\r\n  test-data:\r\n    ec2_az: <availability-zone>\r\n    volumeID: <volume-ID>\r\n    size: 10\r\n```\r\n\r\n**Results:**\r\nThe EBS driver on the Rancher hosts fails to mount the drive, reporting:\r\n```\r\nmount: wrong fs type, bad option, bad superblock on /dev/xvdg,\r\n       missing codepage or helper program, or other error\r\n\r\n       In some cases useful info is found in syslog - try\r\n       dmesg | tail or so.\r\n```\r\nWhen manually connecting to a host and attaching the AWS-created volume to `/dev/xvdg` and attaching the rancher-created volume to `/dev/xvdf`  using the aws-cli, I can reproduce that behavior:\r\n```\r\n$ sudo mount /dev/xvdg <mount-point>\r\nmount: wrong fs type, bad option, bad superblock on /dev/xvdg,\r\n       missing codepage or helper program, or other error\r\n\r\n       In some cases useful info is found in syslog - try\r\n       dmesg | tail or so.\r\n```\r\nInvestigating leads to the following:\r\n```\r\n$ lsblk\r\nNAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT\r\nxvda    202:0    0   8G  0 disk \r\n└─xvda1 202:1    0   8G  0 part /\r\nxvdg    202:96   0  20G  0 disk \r\n└─xvdg1 202:97   0   8G  0 part \r\nxvdf    202:80   0   1G  0 disk /root/rancher_created_volume\r\n```\r\nSo the problem is, that AWS creates a partition on the volume and rancher does not.\r\n\r\nShouldn't the EBS driver support volumes that are created using the de-facto AWS standard?\r\nOr is there a reason not to mount the primary partition?\r\n\r\nIt would really help me in production use-cases, where I don't need to re-create existing volumes and manually copy data there. \r\n\r\nI'm also happy for any workarounds, thanks!"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/405407275","html_url":"https://github.com/rancher/rancher/issues/12106#issuecomment-405407275","issue_url":"https://api.github.com/repos/rancher/rancher/issues/12106","id":405407275,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTQwNzI3NQ==","user":{"login":"StrongMonkey","id":9203378,"node_id":"MDQ6VXNlcjkyMDMzNzg=","avatar_url":"https://avatars0.githubusercontent.com/u/9203378?v=4","gravatar_id":"","url":"https://api.github.com/users/StrongMonkey","html_url":"https://github.com/StrongMonkey","followers_url":"https://api.github.com/users/StrongMonkey/followers","following_url":"https://api.github.com/users/StrongMonkey/following{/other_user}","gists_url":"https://api.github.com/users/StrongMonkey/gists{/gist_id}","starred_url":"https://api.github.com/users/StrongMonkey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StrongMonkey/subscriptions","organizations_url":"https://api.github.com/users/StrongMonkey/orgs","repos_url":"https://api.github.com/users/StrongMonkey/repos","events_url":"https://api.github.com/users/StrongMonkey/events{/privacy}","received_events_url":"https://api.github.com/users/StrongMonkey/received_events","type":"User","site_admin":false},"created_at":"2018-07-16T23:02:13Z","updated_at":"2018-07-16T23:02:13Z","author_association":"MEMBER","body":"@dadarakt we have merged @ahaynssen 's fix into rancher/storage-ebs:v0.9.7 so that you should be able to use an existing EBS volume with partition. If you are using an existing volume, make sure it is formatted as rancher-ebs-driver won't format it for you."}}
{"actor":{"display_login":"StrongMonkey"}}