{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/rkt/issues/3158","repository_url":"https://api.github.com/repos/coreos/rkt","labels_url":"https://api.github.com/repos/coreos/rkt/issues/3158/labels{/name}","comments_url":"https://api.github.com/repos/coreos/rkt/issues/3158/comments","events_url":"https://api.github.com/repos/coreos/rkt/issues/3158/events","html_url":"https://github.com/coreos/rkt/issues/3158","id":174644982,"number":3158,"title":"libvirt on rkt","user":{"login":"Quentin-M","id":1332289,"avatar_url":"https://avatars.githubusercontent.com/u/1332289?v=3","gravatar_id":"","url":"https://api.github.com/users/Quentin-M","html_url":"https://github.com/Quentin-M","followers_url":"https://api.github.com/users/Quentin-M/followers","following_url":"https://api.github.com/users/Quentin-M/following{/other_user}","gists_url":"https://api.github.com/users/Quentin-M/gists{/gist_id}","starred_url":"https://api.github.com/users/Quentin-M/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Quentin-M/subscriptions","organizations_url":"https://api.github.com/users/Quentin-M/orgs","repos_url":"https://api.github.com/users/Quentin-M/repos","events_url":"https://api.github.com/users/Quentin-M/events{/privacy}","received_events_url":"https://api.github.com/users/Quentin-M/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/rkt/labels/dependency/stackanetes","name":"dependency/stackanetes","color":"fbca04"}],"state":"open","locked":false,"assignee":{"login":"iaguis","id":507354,"avatar_url":"https://avatars.githubusercontent.com/u/507354?v=3","gravatar_id":"","url":"https://api.github.com/users/iaguis","html_url":"https://github.com/iaguis","followers_url":"https://api.github.com/users/iaguis/followers","following_url":"https://api.github.com/users/iaguis/following{/other_user}","gists_url":"https://api.github.com/users/iaguis/gists{/gist_id}","starred_url":"https://api.github.com/users/iaguis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iaguis/subscriptions","organizations_url":"https://api.github.com/users/iaguis/orgs","repos_url":"https://api.github.com/users/iaguis/repos","events_url":"https://api.github.com/users/iaguis/events{/privacy}","received_events_url":"https://api.github.com/users/iaguis/received_events","type":"User","site_admin":false},"assignees":[{"login":"iaguis","id":507354,"avatar_url":"https://avatars.githubusercontent.com/u/507354?v=3","gravatar_id":"","url":"https://api.github.com/users/iaguis","html_url":"https://github.com/iaguis","followers_url":"https://api.github.com/users/iaguis/followers","following_url":"https://api.github.com/users/iaguis/following{/other_user}","gists_url":"https://api.github.com/users/iaguis/gists{/gist_id}","starred_url":"https://api.github.com/users/iaguis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iaguis/subscriptions","organizations_url":"https://api.github.com/users/iaguis/orgs","repos_url":"https://api.github.com/users/iaguis/repos","events_url":"https://api.github.com/users/iaguis/events{/privacy}","received_events_url":"https://api.github.com/users/iaguis/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/coreos/rkt/milestones/49","html_url":"https://github.com/coreos/rkt/milestones/v1.15.0","labels_url":"https://api.github.com/repos/coreos/rkt/milestones/49/labels","id":1922997,"number":49,"title":"v1.15.0","description":"","creator":{"login":"lucab","id":98086,"avatar_url":"https://avatars.githubusercontent.com/u/98086?v=3","gravatar_id":"","url":"https://api.github.com/users/lucab","html_url":"https://github.com/lucab","followers_url":"https://api.github.com/users/lucab/followers","following_url":"https://api.github.com/users/lucab/following{/other_user}","gists_url":"https://api.github.com/users/lucab/gists{/gist_id}","starred_url":"https://api.github.com/users/lucab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lucab/subscriptions","organizations_url":"https://api.github.com/users/lucab/orgs","repos_url":"https://api.github.com/users/lucab/repos","events_url":"https://api.github.com/users/lucab/events{/privacy}","received_events_url":"https://api.github.com/users/lucab/received_events","type":"User","site_admin":false},"open_issues":20,"closed_issues":17,"state":"open","created_at":"2016-08-04T07:34:43Z","updated_at":"2016-09-15T09:04:51Z","due_on":"2016-09-15T00:00:00Z","closed_at":null},"comments":0,"created_at":"2016-09-01T22:58:08Z","updated_at":"2016-09-15T09:06:05Z","closed_at":null,"body":"As part of bringing OpenStack on rkt and Kubernetes, the goal is to use libvirtd inside a rkt container and use it to manipulate (e.g. create) VMs on the host. To achieve this, libvirtd [communicates](https://libvirt.org/cgroups.html) via D-Bus with the host's systemd, to create scopes to hold the virtual machines. With the virtual machines living on the host, the libvirtd container can potentially be restarted without influencing the VMs.\r\n\r\nFor these manipulations to function, we have to mount `/sys/fs/cgroup` and `/run`. libvirt also requires access to various other devices.\r\n\r\nThe recent implementation of ```--insecure-options=paths``` (rkt 1.13 - 1.14) now allows us to mount the required paths into rkt containers. However, few issues remain.\r\n\r\n## Experiments\r\n\r\nThe following doesn't represent the real setup as it is fairly more complicated (nova, open vSwitch, ...) but instead focused on starting a KVM VM on the host from a container.\r\n\r\n### With Docker (--pid=host)\r\n\r\nThe way docker operates permis us to do it pretty easily, with only three mounts, `--net=host` and `--pid=host`.\r\n\r\n```\r\n# docker run -it --privileged -v /sys/fs/cgroup:/sys/fs/cgroup -v /run:/run -v /var/lib/libvirt:/var/lib/libvirt --net=host --pid=host quay.io/quentin_m/libvirt\r\n# docker exec -it $(docker ps|grep libvirt|awk '{ print $1 }') bash\r\n$ virsh net-start default\r\n$ virt-install --virt-type kvm --name jessie-docker-3 --location http://httpredir.debian.org/debian/dists/jessie/main/installer-amd64/ --extra-args \"console=ttyS0\" -v --os-variant debiansqueeze --disk size=4 --memory 512\r\n```\r\n\r\n-> Dropped into Debian's installer.\r\n\r\n### With rkt 1.14\r\n\r\n### stage1-coreos\r\n\r\nWith rkt, we need to mount slightly more paths to get libvirt started without any errors.\r\n\r\n```\r\n# rkt run --insecure-options=image,all-run --interactive --net=host \\\r\n  --volume sysfscgroup,kind=host,source=/sys/fs/cgroup --mount volume=sysfscgroup,target=/sys/fs/cgroup \\\r\n  --volume sysclass,kind=host,source=/sys/class --mount volume=sysclass,target=/sys/class \\\r\n  --volume sysdevices,kind=host,source=/sys/devices --mount volume=sysdevices,target=/sys/devices \\\r\n  --volume sysbus,kind=host,source=/sys/bus --mount volume=sysbus,target=/sys/bus \\\r\n  --volume run,kind=host,source=/run --mount volume=run,target=/run \\\r\n  --volume devkvm,kind=host,source=/dev/kvm --mount volume=devkvm,target=/dev/kvm \\\r\n  --volume devmem,kind=host,source=/dev/mem --mount volume=devmem,target=/dev/mem \\\r\n  --volume procsysnet,kind=host,source=/proc/sys/net --mount volume=procsysnet,target=/proc/sys/net \\\r\n  docker://quay.io/quentin_m/libvirt\r\n# rkt enter $(rkt list|grep running|grep libvirt|awk '{print $1}')\r\n# (container): virsh net-start default\r\n# (container) virt-install --virt-type kvm --name jessie-amd64-8 --location http://httpredir.debian.org/debian/dists/jessie/main/installer-amd64/ --extra-args \"console=ttyS0\" -v --os-variant debiansqueeze --disk size=4 --memory 512\r\n2016-09-01 20:59:33.561+0000: 13: warning : qemuOpenVhostNet:505 : Unable to open vhost-net. Opened so far 0, requested 1\r\n2016-09-01 20:59:33.576+0000: 13: error : virDBusCall:1537 : error from service: CreateMachine: Start job for unit machine-qemu\\x2djessie\\x2damd64.scope failed with 'failed'\r\n```\r\n\r\nBeside that dubious network error, the newly created transient systemd scope states:\r\n\r\n```\r\nFailed to add PIDs to scope's control group: Invalid argument\r\nFailed to start Virtual Machine qemu-jessie-amd64-8.\r\n```\r\n\r\nThe [strace](http://x.quentin-machu.fr/etocowawoj.tex) shows that the invalid argument occurs when systemd tries to write to:\r\n\r\n```\r\n/sys/fs/cgroup/systemd/machine.slice/machine-qemu\\\\x2djessie\\\\x2damd64\\\\x2d8.scope/cgroup.procs\r\n```\r\n\r\nAnd in fact, that is the same result that we have while when we try to run libvirt inside Docker without `--pid=host`.\r\n\r\nWhile we were trying to figure out what may cause the issue with @alban, we tried to simply run `systemd-run --scope sleep inf`. The presence of the `--scope` parameter generated the following error on the host's systemd:\r\n\r\n``` \r\nFailed to add PIDs to scope's control group: No such process\r\n```\r\n\r\nOur understanding is that systemd-run (inside the container) forks and asks via D-Bus to systemd (on the host) to move its PID into a new cgroup. But, the pid must be different between the host's point of view and the container one - thus creating that error. However, we note that libvirt interacts with systemd quite [differently](https://github.com/libvirt/libvirt/blob/master/src/util/virsystemd.c) than systemd-run does.\r\n\r\n### stage1-fly\r\n\r\nBecause stage1-coreos doesn't support `--pid=host`, I decided to give stage1-fly a try, by simply swapping the stage1 image:\r\n\r\n```\r\n# /share/bin/rkt-v1.14.0/rkt run --insecure-options=image,all-run --interactive \\\r\n  --stage1-name=coreos.com/rkt/stage1-fly:1.14.0 \\\r\n  --volume sysfscgroup,kind=host,source=/sys/fs/cgroup --mount volume=sysfscgroup,target=/sys/fs/cgroup \\\r\n  --volume sysclass,kind=host,source=/sys/class --mount volume=sysclass,target=/sys/class \\\r\n  --volume sysdevices,kind=host,source=/sys/devices --mount volume=sysdevices,target=/sys/devices \\\r\n  --volume sysbus,kind=host,source=/sys/bus --mount volume=sysbus,target=/sys/bus \\\r\n  --volume run,kind=host,source=/run --mount volume=run,target=/run \\\r\n  --volume devkvm,kind=host,source=/dev/kvm --mount volume=devkvm,target=/dev/kvm \\\r\n  --volume devmem,kind=host,source=/dev/mem --mount volume=devmem,target=/dev/mem \\\r\n  --volume procsysnet,kind=host,source=/proc/sys/net --mount volume=procsysnet,target=/proc/sys/net \\\r\n  --volume varliblibvirt,kind=host,source=/var/lib/libvirt --mount volume=varliblibvirt,target=/var/lib/libvirt \\\r\n  docker://quay.io/quentin_m/libvirt\r\n```\r\n\r\nAt this instant, `/sys/fs/cgroup/systemd` on the host is emptied (unmounted?) and the host is good for a hard reboot. \r\n\r\nIf we try hard by running a second identical rkt fly container to run virt-install, the scope, not surprisingly, fails again to get created, but this time with:\r\n\r\n```\r\nFailed to create cgroup /machine.slice/machine-qemu\\x2djessie\\x2damd64\\x2d10.scope: Read-only file system\r\nFailed to add PIDs to scope's control group: Read-only file system\r\n```\r\n\r\nstrace then shows that systemd failed to re-create `/sys/fs/cgroup/systemd/machine.slice`.\r\n\r\n### stage1-fly (no proc/dev/sys mounts)\r\n\r\nBecause /proc, /dev and /sys are already mounted recursively by rkt-fly, I figured that I should try without specifying these mounts:\r\n\r\n```\r\n# rkt run --insecure-options=image,all-run --interactive \\\r\n  --stage1-name=coreos.com/rkt/stage1-fly:1.14.0 \\\r\n  --volume run,kind=host,source=/run --mount volume=run,target=/run \\\r\n  --volume varliblibvirt,kind=host,source=/var/lib/libvirt --mount volume=varliblibvirt,target=/var/lib/libvirt \\\r\n  docker://quay.io/quentin_m/libvirt\r\n\r\n# rkt run --insecure-options=image,all-run --interactive \\\r\n  --stage1-name=coreos.com/rkt/stage1-fly:1.14.0 \\\r\n  --volume run,kind=host,source=/run --mount volume=run,target=/run \\\r\n  --volume varliblibvirt,kind=host,source=/var/lib/libvirt --mount volume=varliblibvirt,target=/var/lib/libvirt \\\r\n  docker://quay.io/quentin_m/libvirt \\\r\n  --exec=/bin/bash\r\n\r\n# (container) virsh -- net-start default\r\n# (container) virt-install --virt-type kvm --name jessie-amd64-10 --location http://httpredir.debian.org/debian/dists/jessie/main/installer-amd64/ --extra-args \"console=ttyS0\" -v --os-variant debiansqueeze --disk size=4 --memory 512\r\n```\r\n\r\n-> Dropped into Debian's installer\r\n\r\n## Summary\r\n\r\n- `--pid=host` is required,\r\n  - not available with stage1-coreos,\r\n  - by design with stage1-fly,\r\n- specifying the mounts with stage1-fly corrupts the host, we have to omit them: not ideal when the orchestrator has no clue of the underlying runtime - or when we want to simply swap the runtimes,\r\n- stage1-fly [doesn't allow](https://github.com/coreos/rkt/blob/0d72cd886e295d19a007db0dee02aa853a29dcf7/stage1_fly/run/main.go#L258) running multiple containers in one pod, and splitting our nova-compute pod is not really acceptable/conceivabe,\r\n- we don't really want to use fly anyways.\r\n\r\nI am not extremely familiar with systemd's internals - so please bear with me if I am missing something obvious or if I am saying something stupid here.\r\n\r\nThanks."},"comment":{"url":"https://api.github.com/repos/coreos/rkt/issues/comments/247275721","html_url":"https://github.com/coreos/rkt/issues/3158#issuecomment-247275721","issue_url":"https://api.github.com/repos/coreos/rkt/issues/3158","id":247275721,"user":{"login":"s-urbaniak","id":375856,"avatar_url":"https://avatars.githubusercontent.com/u/375856?v=3","gravatar_id":"","url":"https://api.github.com/users/s-urbaniak","html_url":"https://github.com/s-urbaniak","followers_url":"https://api.github.com/users/s-urbaniak/followers","following_url":"https://api.github.com/users/s-urbaniak/following{/other_user}","gists_url":"https://api.github.com/users/s-urbaniak/gists{/gist_id}","starred_url":"https://api.github.com/users/s-urbaniak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/s-urbaniak/subscriptions","organizations_url":"https://api.github.com/users/s-urbaniak/orgs","repos_url":"https://api.github.com/users/s-urbaniak/repos","events_url":"https://api.github.com/users/s-urbaniak/events{/privacy}","received_events_url":"https://api.github.com/users/s-urbaniak/received_events","type":"User","site_admin":false},"created_at":"2016-09-15T09:06:05Z","updated_at":"2016-09-15T09:06:05Z","body":"@iaguis bumping to the next release. We can make a quickfix too once there is a solution."}}
{"actor":{"display_login":"s-urbaniak"}}