{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/3066","labels_url":"https://api.github.com/repos/rancher/rancher/issues/3066/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/3066/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/3066/events","html_url":"https://github.com/rancher/rancher/issues/3066","id":122947003,"number":3066,"title":"Rancher persistant storage (glusterfs+convoy) doesn't work","user":{"login":"teon","id":139682,"avatar_url":"https://avatars.githubusercontent.com/u/139682?v=3","gravatar_id":"","url":"https://api.github.com/users/teon","html_url":"https://github.com/teon","followers_url":"https://api.github.com/users/teon/followers","following_url":"https://api.github.com/users/teon/following{/other_user}","gists_url":"https://api.github.com/users/teon/gists{/gist_id}","starred_url":"https://api.github.com/users/teon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/teon/subscriptions","organizations_url":"https://api.github.com/users/teon/orgs","repos_url":"https://api.github.com/users/teon/repos","events_url":"https://api.github.com/users/teon/events{/privacy}","received_events_url":"https://api.github.com/users/teon/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rancher/rancher/labels/area/storage","name":"area/storage","color":"fbca04"},{"url":"https://api.github.com/repos/rancher/rancher/labels/kind/bug","name":"kind/bug","color":"fc2929"},{"url":"https://api.github.com/repos/rancher/rancher/labels/release/v1.0.0","name":"release/v1.0.0","color":"bfdadc"}],"state":"open","locked":false,"assignee":{"login":"cjellick","id":2473240,"avatar_url":"https://avatars.githubusercontent.com/u/2473240?v=3","gravatar_id":"","url":"https://api.github.com/users/cjellick","html_url":"https://github.com/cjellick","followers_url":"https://api.github.com/users/cjellick/followers","following_url":"https://api.github.com/users/cjellick/following{/other_user}","gists_url":"https://api.github.com/users/cjellick/gists{/gist_id}","starred_url":"https://api.github.com/users/cjellick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cjellick/subscriptions","organizations_url":"https://api.github.com/users/cjellick/orgs","repos_url":"https://api.github.com/users/cjellick/repos","events_url":"https://api.github.com/users/cjellick/events{/privacy}","received_events_url":"https://api.github.com/users/cjellick/received_events","type":"User","site_admin":false},"milestone":null,"comments":8,"created_at":"2015-12-18T13:18:33Z","updated_at":"2016-02-01T20:03:10Z","closed_at":null,"body":"I have spent literally few days debugging the problems, and here is a long list of errors I have found - and now I'm unable to do more - rest is up to you guys.\r\nFirst of all I'm going to write about catalog stack errors - which later I have overcome/fixed by copying the rancher/docker-compose files and fixing them by hand - and then deploying using rancher-compose command line tool.\r\n\r\n## GlusterFS catalog stack errors\r\n\r\nFirst of all GlusterFS deploys all servers/volumes on one host. I have fixed this by adding `io.rancher.scheduler.global: 'true'` - the server and sidekicks were deployed to all my hosts (2). Here are logs after deployment:\r\n\r\n```glusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.543477813Z ame: unrecognized option '--first=yes'\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.578227986Z ame: unrecognized option '--version=1'\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.578262708Z ame: unrecognized option '--volume-op=start'\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.578282845Z ame: unrecognized option '--gd-workdir=/var/lib/glusterd'\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.578304656Z ame: unrecognized option '--first=yes'\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.578327600Z ame: unrecognized option '--version=1'\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.578347693Z ame: unrecognized option '--volume-op=start'\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.578368180Z /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh: line 105: [: =: unary operator expected\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.578390000Z /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh: line 55: smbd: command not found\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.578410578Z ame: unrecognized option '--first=yes'\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.578430155Z ame: unrecognized option '--version=1'\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.578449072Z ame: unrecognized option '--volume-op=start'\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:24:52.567949849Z Samba is not installed\r\nglusterfs-server_glusterfs-peer_1 | 2015-12-18T12:25:52.659501357Z clnt_create: RPC: Timed out\r\nglusterfs-server_glusterfs-volume-create_1 | 2015-12-18T12:22:56.466070480Z Waiting for all service containers to start...\r\nglusterfs-server_glusterfs-volume-create_1 | 2015-12-18T12:23:29.410538066Z Containers are starting...\r\nglusterfs-server_glusterfs-volume-create_1 | 2015-12-18T12:23:29.410740320Z Waiting for Gluster Daemons to come up\r\nglusterfs-server_glusterfs-volume-create_1 | 2015-12-18T12:24:03.926743599Z gluster peer probe 10.42.8.4\r\nglusterfs-server_glusterfs-volume-create_1 | 2015-12-18T12:24:10.560995830Z peer probe: success.\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:22:56.978711108Z Waiting for all containers to come up...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:23:29.927440438Z Containers are coming up...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:23:30.383645924Z Waiting for pool...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:23:35.409371020Z Waiting for pool...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:23:40.440145270Z Waiting for pool...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:23:45.512919994Z Waiting for pool...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:23:50.491977558Z Waiting for pool...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:23:55.517773525Z Waiting for pool...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:24:00.540789027Z Waiting for pool...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:24:05.558649872Z pool list: failed\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:24:05.561807499Z Waiting for pool...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:24:10.584400817Z Waiting for peerprobes and gluster daemons to come on line\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:24:45.645693153Z Getting peer mount points...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:24:46.148033310Z Volume production does not exist\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:24:46.150243759Z Creating volume production...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:24:46.709890388Z volume create: production: success: please start the volume to access data\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:24:51.741458641Z Starting volume production...\r\nglusterfs-server_glusterfs-data_2          | 2015-12-18T12:24:53.296710878Z volume start: production: success\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.868732186Z ame: unrecognized option '--first=yes'\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.868842879Z ame: unrecognized option '--version=1'\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.868855053Z ame: unrecognized option '--volume-op=start'\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.868865219Z ame: unrecognized option '--gd-workdir=/var/lib/glusterd'\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.873645534Z ame: unrecognized option '--first=yes'\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.873667200Z ame: unrecognized option '--version=1'\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.873676700Z ame: unrecognized option '--volume-op=start'\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.878880228Z /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh: line 105: [: =: unary operator expected\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.879672567Z /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh: line 55: smbd: command not found\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.881212950Z Samba is not installed\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.888321726Z ame: unrecognized option '--first=yes'\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.888480469Z ame: unrecognized option '--version=1'\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:27:42.888495993Z ame: unrecognized option '--volume-op=start'\r\nglusterfs-server_glusterfs-peer_2          | 2015-12-18T12:28:42.951284983Z clnt_create: RPC: Timed out\r\nglusterfs-server_glusterfs-volume-create_2 | 2015-12-18T12:26:32.264327047Z Waiting for all service containers to start...\r\nglusterfs-server_glusterfs-volume-create_2 | 2015-12-18T12:26:32.578568582Z Containers are starting...\r\nglusterfs-server_glusterfs-volume-create_2 | 2015-12-18T12:26:32.578710093Z Waiting for Gluster Daemons to come up\r\nglusterfs-server_glusterfs-volume-create_2 | 2015-12-18T12:26:32.834113358Z Waiting for all containers to come up...\r\nglusterfs-server_glusterfs-volume-create_2 | 2015-12-18T12:26:33.149553666Z Containers are coming up...\r\nglusterfs-server_glusterfs-volume-create_2 | 2015-12-18T12:26:33.505792447Z The lowest numbered container handles volume operations... I'm not the lowest```\r\n\r\nThe funny thing is that after deploying by hand with rancher compose - all services/containers are green/running, but the *Stack status is Degraded*...\r\n\r\n## Convoy Gluster catalog stack errors\r\n\r\nWhen launching Convoy Gluster from catalog:\r\n1. the storagepoll container doesn't start - I need to add to all my hosts labels `convoy.glusterfs=true` in order to start the storagepoll container\r\n2. it doens't start on hosts where glusterfs server is running.\r\n\r\nI also needed to copy and deploy by hand with label `io.rancher.scheduler.global: 'true'` in order to run convoy on all my storage hosts.\r\n\r\nHere are my logs of convoy stack:\r\n\r\n```\r\nonvoy-gluster-storagepool_2 | 2015-12-18T12:31:05.117702584Z time=\"2015-12-18T12:31:05Z\" level=info msg=\"Listening for health checks on 0.0.0.0:10241/healthcheck\"\r\nconvoy-gluster-storagepool_2 | 2015-12-18T12:31:10.439820922Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"storagepool event [cf18cb7d-10cb-416a-901b-f84b4b89fde1 054af08c-1a53-44a6-b0a9-2b469a2eeb68]\"\r\nconvoy-gluster_2             | 2015-12-18T12:31:08.891511812Z Waiting for metadata.time=\"2015-12-18T12:31:08Z\" level=info msg=\"Execing [/usr/bin/nsenter --mount=/proc/11768/ns/mnt -F -- /srv/docker/aufs/mnt/cad551b2df487b2ddf5fdef48282f748305e4e1c94614e25ac311e5fb1901b08/var/lib/rancher/convoy-agent/share-mnt --stage2 /var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466 -- /launch volume-agent-glusterfs-internal]\"\r\nconvoy-gluster_2             | 2015-12-18T12:31:09.085481084Z Registering convoy socket at /var/run/conoy-convoy.sock\r\nconvoy-gluster_2             | 2015-12-18T12:31:09.090363388Z time=\"2015-12-18T12:31:09Z\" level=info msg=\"Listening for health checks on 0.0.0.0:10241/healthcheck\"\r\nconvoy-gluster_2             | 2015-12-18T12:31:09.090495009Z time=\"2015-12-18T12:31:09Z\" level=info msg=\"Got: driver-opts [glusterfs.defaultvolumepool=production glusterfs.servers=glusterfs]\"\r\nconvoy-gluster_2             | 2015-12-18T12:31:09.090564830Z time=\"2015-12-18T12:31:09Z\" level=info msg=\"Got: root /var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466\"\r\nconvoy-gluster_2             | 2015-12-18T12:31:09.090583492Z time=\"2015-12-18T12:31:09Z\" level=info msg=\"Got: drivers [glusterfs]\"\r\nconvoy-gluster_2             | 2015-12-18T12:31:09.090817296Z time=\"2015-12-18T12:31:09Z\" level=info msg=\"Launching convoy with args: [--socket=/host/var/run/conoy-convoy.sock daemon --driver-opts=glusterfs.defaultvolumepool=production --driver-opts=glusterfs.servers=glusterfs --root=/var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466 --drivers=glusterfs]\"\r\nconvoy-gluster_2             | 2015-12-18T12:31:09.133825733Z time=\"2015-12-18T12:31:09Z\" level=debug msg=\"Creating config at /var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:09.133922522Z time=\"2015-12-18T12:31:09Z\" level=debug msg= driver=glusterfs driver_opts=map[glusterfs.defaultvolumepool:production glusterfs.servers:glusterfs] event=init pkg=daemon reason=prepare root=\"/var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466\"\r\nconvoy-gluster_2             | 2015-12-18T12:31:09.461668443Z time=\"2015-12-18T12:31:09Z\" level=debug msg=\"Volume production is being mounted it to /var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466/glusterfs/mounts/production, with option [-t glusterfs]\" pkg=util\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.231985544Z time=\"2015-12-18T12:31:10Z\" level=error msg=\"Get http:///host/var/run/conoy-convoy.sock/v1/volumes/list: dial unix /host/var/run/conoy-convoy.sock: connection refused\"\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.829010660Z time=\"2015-12-18T12:31:10Z\" level=debug msg= driver=glusterfs event=init pkg=daemon reason=complete\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.832262893Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering GET, /uuid\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.832442628Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering GET, /volumes/list\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.832753585Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering GET, /volumes/\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.832998271Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering GET, /snapshots/\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.833202060Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering GET, /backups/list\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.834584845Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering GET, /backups/inspect\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.834801289Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering GET, /info\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.834998475Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering POST, /backups/create\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.835369498Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering POST, /volumes/create\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.835549177Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering POST, /volumes/mount\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.835902847Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering POST, /volumes/umount\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.836200048Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering POST, /snapshots/create\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.836372348Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering DELETE, /snapshots/\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.836612838Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering DELETE, /backups\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.838068642Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering DELETE, /volumes/\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.838313649Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering plugin handler POST, /VolumeDriver.Unmount\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.838566051Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering plugin handler POST, /VolumeDriver.Path\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.838701301Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering plugin handler POST, /Plugin.Activate\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.838808310Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering plugin handler POST, /VolumeDriver.Create\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.838934385Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering plugin handler POST, /VolumeDriver.Remove\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.839131453Z time=\"2015-12-18T12:31:10Z\" level=debug msg=\"Registering plugin handler POST, /VolumeDriver.Mount\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:31:10.839305008Z time=\"2015-12-18T12:31:10Z\" level=warning msg=\"Remove previous sockfile at /host/var/run/conoy-convoy.sock\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:01.675015517Z Waiting for metadata.time=\"2015-12-18T12:34:01Z\" level=info msg=\"Execing [/usr/bin/nsenter --mount=/proc/7012/ns/mnt -F -- /srv/docker/aufs/mnt/55c457c1b57b751f6a5d5f6cd495b773376df56b20439d379c9b1abf72c8e333/var/lib/rancher/convoy-agent/share-mnt --stage2 /var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466 -- /launch volume-agent-glusterfs-internal]\"\r\nconvoy-gluster_2             | 2015-12-18T12:34:01.793155165Z Registering convoy socket at /var/run/conoy-convoy.sock\r\nconvoy-gluster_2             | 2015-12-18T12:34:01.796432368Z time=\"2015-12-18T12:34:01Z\" level=info msg=\"Listening for health checks on 0.0.0.0:10241/healthcheck\"\r\nconvoy-gluster_2             | 2015-12-18T12:34:01.796526868Z time=\"2015-12-18T12:34:01Z\" level=info msg=\"Got: root /var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466\"\r\nconvoy-gluster_2             | 2015-12-18T12:34:01.796706666Z time=\"2015-12-18T12:34:01Z\" level=info msg=\"Got: drivers [glusterfs]\"\r\nconvoy-gluster_2             | 2015-12-18T12:34:01.796783515Z time=\"2015-12-18T12:34:01Z\" level=info msg=\"Got: driver-opts [glusterfs.defaultvolumepool=production glusterfs.servers=glusterfs]\"\r\nconvoy-gluster_2             | 2015-12-18T12:34:01.796984871Z time=\"2015-12-18T12:34:01Z\" level=info msg=\"Launching convoy with args: [--socket=/host/var/run/conoy-convoy.sock daemon --root=/var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466 --drivers=glusterfs --driver-opts=glusterfs.defaultvolumepool=production --driver-opts=glusterfs.servers=glusterfs]\"\r\nconvoy-gluster_2             | 2015-12-18T12:34:01.802177041Z time=\"2015-12-18T12:34:01Z\" level=debug msg=\"Creating config at /var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:01.802384085Z time=\"2015-12-18T12:34:01Z\" level=debug msg= driver=glusterfs driver_opts=map[glusterfs.servers:glusterfs glusterfs.defaultvolumepool:production] event=init pkg=daemon reason=prepare root=\"/var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466\"\r\nconvoy-gluster_2             | 2015-12-18T12:34:01.942347031Z time=\"2015-12-18T12:34:01Z\" level=debug msg=\"Volume production is being mounted it to /var/lib/rancher/convoy/convoy-d719d5c7-1564-4174-bedc-858d33958466/glusterfs/mounts/production, with option [-t glusterfs]\" pkg=util\r\nconvoy-gluster_2             | 2015-12-18T12:34:02.920596731Z time=\"2015-12-18T12:34:02Z\" level=error msg=\"Get http:///host/var/run/conoy-convoy.sock/v1/volumes/list: dial unix /host/var/run/conoy-convoy.sock: connection refused\"\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.058882367Z time=\"2015-12-18T12:34:03Z\" level=debug msg= driver=glusterfs event=init pkg=daemon reason=complete\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.060713470Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering GET, /uuid\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.061430451Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering GET, /volumes/list\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.061573468Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering GET, /volumes/\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.061723172Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering GET, /snapshots/\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.061872977Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering GET, /backups/list\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.062035761Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering GET, /backups/inspect\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.062284233Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering GET, /info\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.064996729Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering POST, /volumes/create\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065020461Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering POST, /volumes/mount\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065029593Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering POST, /volumes/umount\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065038224Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering POST, /snapshots/create\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065046533Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering POST, /backups/create\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065054801Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering DELETE, /backups\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065062820Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering DELETE, /volumes/\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065070961Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering DELETE, /snapshots/\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065079403Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering plugin handler POST, /VolumeDriver.Mount\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065088025Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering plugin handler POST, /VolumeDriver.Unmount\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065096326Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering plugin handler POST, /VolumeDriver.Path\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065104632Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering plugin handler POST, /Plugin.Activate\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065112813Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering plugin handler POST, /VolumeDriver.Create\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065121183Z time=\"2015-12-18T12:34:03Z\" level=debug msg=\"Registering plugin handler POST, /VolumeDriver.Remove\" pkg=daemon\r\nconvoy-gluster_2             | 2015-12-18T12:34:03.065129348Z time=\"2015-12-18T12:34:03Z\" level=warning msg=\"Remove previous sockfile at /host/var/run/conoy-convoy.sock\" pkg=daemon\r\nconvoy-gluster-storagepool_2 | 2015-12-18T12:33:57.028327576Z time=\"2015-12-18T12:33:57Z\" level=info msg=\"Listening for health checks on 0.0.0.0:10241/healthcheck\"\r\nconvoy-gluster-storagepool_2 | 2015-12-18T12:34:02.296076437Z time=\"2015-12-18T12:34:02Z\" level=debug msg=\"storagepool event [054af08c-1a53-44a6-b0a9-2b469a2eeb68 cf18cb7d-10cb-416a-901b-f84b4b89fde1]\"\r\n```\r\n\r\n## Convoy-Gluster is not available\r\n\r\nNow deploying an application using `volume_driver: convoy-gluster` - no luck, here are the errors from rancher itself:\r\n\r\n```\r\n2015-12-18 13:15:19,066 ERROR [0fb1c55d-9cae-43f5-a74d-99dded196391:1617] [instance:84->instanceHostMap:84] [instance.start->(InstanceStart)->instancehostmap.activate] [] [utorService-105] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception io.cattle.platform.eventing.exception.EventExecutionException: 19cb44d3-0a73-4550-8c09-4df3252786c0 : 500 Server Error: Internal Server Error (\"Error looking up volume plugin convoy-gluster: Post http:///var/run/conoy-convoy-gluster.sock/Plugin.Activate: dial unix /var/run/conoy-convoy-gluster.sock: connection refused\")\r\n\tat io.cattle.platform.eventing.exception.EventExecutionException.fromEvent(EventExecutionException.java:53) ~[cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.agent.impl.RemoteAgentImpl.callSync(RemoteAgentImpl.java:72) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.agent.impl.RemoteAgentImpl.callSync(RemoteAgentImpl.java:158) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.common.handler.AgentBasedProcessHandler.callSync(AgentBasedProcessHandler.java:170) ~[cattle-iaas-logic-common-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.common.handler.AgentBasedProcessHandler.handleEvent(AgentBasedProcessHandler.java:156) ~[cattle-iaas-logic-common-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.common.handler.AgentBasedProcessHandler.handle(AgentBasedProcessHandler.java:101) ~[cattle-iaas-logic-common-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:456) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:396) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:390) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:390) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:520) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:323) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:259) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:256) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:188) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:161) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:110) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:107) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.object.process.impl.DefaultObjectProcessManager.executeStandardProcess(DefaultObjectProcessManager.java:29) [cattle-framework-object-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.common.handler.AbstractObjectProcessLogic.activate(AbstractObjectProcessLogic.java:31) [cattle-iaas-logic-common-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.instance.InstanceStart.compute(InstanceStart.java:220) [cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.process.instance.InstanceStart.handle(InstanceStart.java:85) [cattle-iaas-logic-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:456) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:396) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:390) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:390) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:520) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:323) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:259) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:256) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:188) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:161) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:110) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:107) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:107) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.engine.eventing.impl.ProcessEventListenerImpl.processExecute(ProcessEventListenerImpl.java:66) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]\r\n\tat sun.reflect.GeneratedMethodAccessor450.invoke(Unknown Source) ~[na:na]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_85]\r\n\tat java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_85]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener$1.doWithLockNoResult(MethodInvokingListener.java:76) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.annotation.MethodInvokingListener.onEvent(MethodInvokingListener.java:72) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat io.cattle.platform.eventing.impl.AbstractThreadPoolingEventService$2.doRun(AbstractThreadPoolingEventService.java:135) [cattle-framework-eventing-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:13) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_85]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_85]\r\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_85]\r\n```\r\n\r\nSo after few days I'm giving up ;-("},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/178164242","html_url":"https://github.com/rancher/rancher/issues/3066#issuecomment-178164242","issue_url":"https://api.github.com/repos/rancher/rancher/issues/3066","id":178164242,"user":{"login":"deniseschannon","id":11410997,"avatar_url":"https://avatars.githubusercontent.com/u/11410997?v=3","gravatar_id":"","url":"https://api.github.com/users/deniseschannon","html_url":"https://github.com/deniseschannon","followers_url":"https://api.github.com/users/deniseschannon/followers","following_url":"https://api.github.com/users/deniseschannon/following{/other_user}","gists_url":"https://api.github.com/users/deniseschannon/gists{/gist_id}","starred_url":"https://api.github.com/users/deniseschannon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deniseschannon/subscriptions","organizations_url":"https://api.github.com/users/deniseschannon/orgs","repos_url":"https://api.github.com/users/deniseschannon/repos","events_url":"https://api.github.com/users/deniseschannon/events{/privacy}","received_events_url":"https://api.github.com/users/deniseschannon/received_events","type":"User","site_admin":false},"created_at":"2016-02-01T20:03:10Z","updated_at":"2016-02-01T20:03:10Z","body":"We are preparing a new glusterfs template that should be out in the next couple of weeks that should help resolve a lot of these issues. "}}