{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/696","repository_url":"https://api.github.com/repos/coreos/coreos-kubernetes","labels_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/696/labels{/name}","comments_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/696/comments","events_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/696/events","html_url":"https://github.com/coreos/coreos-kubernetes/issues/696","id":179897399,"number":696,"title":"glusterfs mount on coreos using special docker container - problem with monitoring in kubernetes","user":{"login":"Rastusik","id":1735097,"avatar_url":"https://avatars.githubusercontent.com/u/1735097?v=3","gravatar_id":"","url":"https://api.github.com/users/Rastusik","html_url":"https://github.com/Rastusik","followers_url":"https://api.github.com/users/Rastusik/followers","following_url":"https://api.github.com/users/Rastusik/following{/other_user}","gists_url":"https://api.github.com/users/Rastusik/gists{/gist_id}","starred_url":"https://api.github.com/users/Rastusik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Rastusik/subscriptions","organizations_url":"https://api.github.com/users/Rastusik/orgs","repos_url":"https://api.github.com/users/Rastusik/repos","events_url":"https://api.github.com/users/Rastusik/events{/privacy}","received_events_url":"https://api.github.com/users/Rastusik/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-09-28T21:39:20Z","updated_at":"2016-09-30T14:23:23Z","closed_at":null,"body":"Hi, I'm not sure whether this is the right place to post the problem, but the main problem is kubernetes running specifically on coreos, so I'll give it a try.\r\n\r\nI was able to mount a glusterfs filesystem into coreos using this guide: https://huaminchen.wordpress.com/2015/06/09/share-hosts-mount-namespace-with-docker-containers/ with some modifications - I needed to use shared volume mounts with docker.\r\n\r\nTo be able to run the whole thing, I needed to change the MountFlags parameter in the systemd config of docker.service to 'shared'\r\n\r\n```\r\nMountFlags=shared\r\n```\r\n\r\nThen I was able to run this docker image to be able to mount the glusterfs from within the docker container:\r\n\r\n```\r\ndocker run --privileged --net=host  -v /:/host:shared -i -t centos bash\r\n```\r\n\r\nFrom within the container I just installed the glusterfs tools and mounted the glusterfs volume onto a directory from the /host path, so it would be usable from within the VM.\r\n\r\nThat actually worked, but with 2 drawbacks:\r\n\r\n1) I'm not able to stop the container, even after manually unmounting the glusterfs volume from within the container, the process just hangs. To be able to clearly unmount, I need to stop kubelet and docker in coreos and restart the vhole VM (btw dockker stops at least a minute this way).. this is a problem but I can somehow live with it\r\n\r\n2) I am able to use the mount point in kubernetes pods using hostpath volumes, BUT: there is a problem with getting monitoring data from the kubernetes heapster pod into the influxdb-grafana pod (no idea why, I use the standard config from kubernetes repo) - node data is being collected from every node, but there is no new data from the running pods - it's not being written, even the pod names of the newly created pods are missing... I think there is also a problem with writting logs into the ES-Kibana pods (also standard setup from the kube repo), but at first I need to solve the monitoring.\r\n\r\n\r\nThis happens when I run the mounter docker on coreos start slightly before the kubelet process starts. I realized, that when I run the mounter docker after kubelet process starts and after some basic kubernetes containers are online - in this cas the monitoring data of the pods is being logged from heapster into influxdb. I suppose there is some magic being done with the mounts after kubelet starts, but I have no idea what that is and what to do about, and I also have no idea how to run the mounter docker after kubelet, but after all the mounting magic happens (I was also able to run the mounter right after kubelet started, using systemd ExecAfter, but it still was too soon for some reason...)\r\n\r\nCould anybody help me understand what's under the hood or maybe suggest a solution of the problem? Maybe there's a cleaner way to mount glusterfs on coreos, but I haven't found any. Is there? This solution works almost perfectly, but still just 95% good... I just realized that after a week of usage, everything worked like a charm, except monitoring of the pods and some logging.\r\n\r\nI'm using the current stable version of coreos - 1222.2.0, and kubernetes 1.3.5 (v1.3.5+coreos.0)"},"comment":{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/comments/250757368","html_url":"https://github.com/coreos/coreos-kubernetes/issues/696#issuecomment-250757368","issue_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/696","id":250757368,"user":{"login":"Rastusik","id":1735097,"avatar_url":"https://avatars.githubusercontent.com/u/1735097?v=3","gravatar_id":"","url":"https://api.github.com/users/Rastusik","html_url":"https://github.com/Rastusik","followers_url":"https://api.github.com/users/Rastusik/followers","following_url":"https://api.github.com/users/Rastusik/following{/other_user}","gists_url":"https://api.github.com/users/Rastusik/gists{/gist_id}","starred_url":"https://api.github.com/users/Rastusik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Rastusik/subscriptions","organizations_url":"https://api.github.com/users/Rastusik/orgs","repos_url":"https://api.github.com/users/Rastusik/repos","events_url":"https://api.github.com/users/Rastusik/events{/privacy}","received_events_url":"https://api.github.com/users/Rastusik/received_events","type":"User","site_admin":false},"created_at":"2016-09-30T14:23:23Z","updated_at":"2016-09-30T14:23:23Z","body":"yes, I was also considering this idea, but I haven't found any information regarding how to do it ... or it more complicated than the hostpath solution... could you show me an example of the rbdmap command aliasing? I could try to reproduce it with glusterfs"}}
{"actor":{"display_login":"Rastusik"}}