{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57345","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57345/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57345/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57345/events","html_url":"https://github.com/kubernetes/kubernetes/issues/57345","id":282988420,"node_id":"MDU6SXNzdWUyODI5ODg0MjA=","number":57345,"title":"Number of \"loaded inactive dead\" systemd transient mount units continues to grow","user":{"login":"saad-ali","id":10052848,"node_id":"MDQ6VXNlcjEwMDUyODQ4","avatar_url":"https://avatars1.githubusercontent.com/u/10052848?v=4","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"node_id":"MDU6TGFiZWwxMDUxNDYwNzE=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":169428334,"node_id":"MDU6TGFiZWwxNjk0MjgzMzQ=","url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/storage","name":"sig/storage","color":"d2b48c","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":34,"created_at":"2017-12-18T19:08:44Z","updated_at":"2018-08-06T18:06:01Z","closed_at":"2018-01-19T19:41:06Z","author_association":"MEMBER","body":"<!-- This form is for bug reports and feature requests ONLY! \r\n\r\nIf you're looking for help check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes) and the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).\r\n-->\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?**:\r\n\r\n> Uncomment only one, leave it on its own line: \r\n>\r\n/kind bug\r\n\r\n**What happened**:\r\n\r\nThe number of systemd transient mount units continues to grow unchecked on nodes.\r\n\r\nI see a massive number of `loaded    inactive dead` secret transient mount units, for example: \r\n```\r\nhome-kubernetes-containerized_mounter-rootfs-var-lib-kubelet-pods-947b5769\\x2de3d0\\x2d11e7\\x2d874f\\x2d42010a8002d1-volumes-kubernetes.io\\x7e...-default\\x2dtoken\\x2dvfqcv.mount loaded    inactive dead      /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/94f62320-e319-11e7-874f-42010a8002d1/volumes/kubernetes.io~secret/default-token-vfqcv\r\n```\r\n\r\nWe suspect (but have not yet verified) that once there are too many transient mount units, subsequent mounts will fail with:\r\n```\r\nWarning FailedMount 28m (x40 over 1h) kubelet, ... MountVolume.SetUp failed for volume \"myvol\" : mount failed: exit status 1 \r\nMounting command: systemd-run \r\nMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/8d3db903-e053-11e7-b0ce-42010a800024/volumes/kubernetes.io~secret/myvol --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/8d3db903-e053-11e7-b0ce-42010a800024/volumes/kubernetes.io~secret/myvol \r\nOutput: Failed to start transient scope unit: Argument list too long\r\n```\r\n\r\n**What you expected to happen**:\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\n\r\nI left my one node cluster running over the weekend with a single cron job (based on the example cronjob in https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/).\r\n\r\nOn my test machine I saw the following on Friday evening:\r\n\r\n```SH\r\n$ mount | grep -i kube | wc -l\r\n37\r\n$ ls -l /run/systemd/transient/ | wc -l\r\n1\r\n$ systemctl list-units --all | wc -l\r\n825\r\n```\r\n\r\nAnd the following on Monday morning:\r\n```SH\r\n$ mount | grep -i kube | wc -l\r\n37\r\n$ ls -l /run/systemd/transient/ | wc -l\r\n1\r\n$ systemctl list-units --all | wc -l\r\n16154\r\n```\r\n\r\n\r\n**Anything else we need to know?**:\r\n\r\nSuspect PR https://github.com/kubernetes/kubernetes/pull/49640 which forces mounts to run in their own systemd scope. It went in to k8s 1.8, so all versions of 1.8+ running systemd may be affected.\r\n\r\nCC @jsafrane @derekwaynecarr\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\n- Cloud provider or hardware configuration:\r\n- OS (e.g. from /etc/os-release):\r\n- Kernel (e.g. `uname -a`):\r\n- Install tools:\r\n- Others:\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/410798747","html_url":"https://github.com/kubernetes/kubernetes/issues/57345#issuecomment-410798747","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57345","id":410798747,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMDc5ODc0Nw==","user":{"login":"msau42","id":24448061,"node_id":"MDQ6VXNlcjI0NDQ4MDYx","avatar_url":"https://avatars3.githubusercontent.com/u/24448061?v=4","gravatar_id":"","url":"https://api.github.com/users/msau42","html_url":"https://github.com/msau42","followers_url":"https://api.github.com/users/msau42/followers","following_url":"https://api.github.com/users/msau42/following{/other_user}","gists_url":"https://api.github.com/users/msau42/gists{/gist_id}","starred_url":"https://api.github.com/users/msau42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msau42/subscriptions","organizations_url":"https://api.github.com/users/msau42/orgs","repos_url":"https://api.github.com/users/msau42/repos","events_url":"https://api.github.com/users/msau42/events{/privacy}","received_events_url":"https://api.github.com/users/msau42/received_events","type":"User","site_admin":false},"created_at":"2018-08-06T18:06:01Z","updated_at":"2018-08-06T18:06:01Z","author_association":"MEMBER","body":"@honnix this is fixed in gke 1.9.3+"}}
{"actor":{"display_login":"msau42"}}