{"action":"closed","issue":{"url":"https://api.github.com/repos/cloudfoundry/guardian/issues/53","repository_url":"https://api.github.com/repos/cloudfoundry/guardian","labels_url":"https://api.github.com/repos/cloudfoundry/guardian/issues/53/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry/guardian/issues/53/comments","events_url":"https://api.github.com/repos/cloudfoundry/guardian/issues/53/events","html_url":"https://github.com/cloudfoundry/guardian/issues/53","id":181960590,"number":53,"title":"container leak","user":{"login":"jsievers","id":178022,"avatar_url":"https://avatars.githubusercontent.com/u/178022?v=3","gravatar_id":"","url":"https://api.github.com/users/jsievers","html_url":"https://github.com/jsievers","followers_url":"https://api.github.com/users/jsievers/followers","following_url":"https://api.github.com/users/jsievers/following{/other_user}","gists_url":"https://api.github.com/users/jsievers/gists{/gist_id}","starred_url":"https://api.github.com/users/jsievers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsievers/subscriptions","organizations_url":"https://api.github.com/users/jsievers/orgs","repos_url":"https://api.github.com/users/jsievers/repos","events_url":"https://api.github.com/users/jsievers/events{/privacy}","received_events_url":"https://api.github.com/users/jsievers/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cloudfoundry/guardian/labels/unscheduled","name":"unscheduled","color":"e4eff7"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-10-10T08:31:32Z","updated_at":"2016-10-19T10:04:16Z","closed_at":"2016-10-19T10:04:16Z","body":"\r\n\r\n## Description\r\n\r\nsince some weeks we are seeing concourse containers not being reaped. \r\nConcourse 2.1.0, garden-runc/0.8.0, bosh-openstack-kvm-ubuntu-trusty-go_agent/3263.3\r\nThe symptom is `insufficient subnets remaining in the pool` similar to https://github.com/concourse/concourse/issues/293\r\nIt takes several days for the leak to reach the state where workers run out of containers (250 containers per worker)\r\nThe effect can be seen when comparing `fly containers` with `fly workers`:   `fly workers` shows much more containers than fly containers.\r\nWhen you `bosh ssh` into the worker and use `gaol list`, you can see the \"zombie\" containers.\r\nTrying to `gaol shell` into a \"zombie\" container gives an error\r\n\r\n```\r\nroot@7402fe78-8b6d-4c6e-97fb-e809337980ea:~# /tmp/gaol shell fdd75cce-c0ae-46ee-729b-6f9374525ab9\r\nerror: hijack: Backend error: Exit status: 500, message: {\"Type\":\"\",\"Message\":\"unable to find user root: no matching entries in passwd file\",\"Handle\":\"\"}\r\n```\r\n\r\ngaol properties of a \"dead\" container looks like this:\r\n\r\n```\r\ngarden.grace-time       300000000000\r\ngarden.network.host-ip  10.254.0.93\r\nkawasaki.container-interface    wui37gmn4uj7-1\r\nkawasaki.bridge-interface       wbrdg-0afe005c\r\nkawasaki.dns-servers\r\nkawasaki.mtu    1500\r\nconcourse:volume-mounts {\"2b3475a7-1e37-42c5-7a4b-e49e68074495\":\"/tmp/build/get\"}\r\ngarden.network.container-ip     10.254.0.94\r\ngarden.network.external-ip      10.1.6.17\r\ngarden.state    created\r\nconcourse:resource-result       {\r\n  \"version\": {\r\n    \"digest\": \"sha256:4e9752e8f15bff07872664e7e206678e45dd84bccce02383f9b29a2bd1501864\"\r\n  },\r\n  \"metadata\": [\r\n    {\r\n      \"name\": \"image\",\r\n      \"value\": \"sha256:4c07d\"\r\n    }\r\n  ]\r\n}\r\n\r\nkawasaki.host-interface wui37gmn4uj7-0\r\nkawasaki.iptable-prefix w--\r\n```\r\n\r\n\r\nso it looks like all zombie containers are concourse:resource-result containers, i.e. containers which should have done a git clone.\r\n\r\n## Logging and/or test output\r\n\r\nThe last log entries of a \"dead\" container:\r\n\r\n```\r\n{\"timestamp\":\"1475624427.399276257\",\"source\":\"guardian\",\"message\":\"guardian.run.exec.finished\",\"log_level\":1,\"data\":{\"handle\":\"a9645f41-a32f-4fa7-58e4-98838a3191d8\",\"id\":\"a9645f41-a32f-4fa7-58e4-98838a3191d8\",\"path\":\"/opt/resource/in\",\"session\":\"6796.2\"}}\r\n{\"timestamp\":\"1475624427.399291754\",\"source\":\"guardian\",\"message\":\"guardian.run.finished\",\"log_level\":1,\"data\":{\"handle\":\"a9645f41-a32f-4fa7-58e4-98838a3191d8\",\"path\":\"/opt/resource/in\",\"session\":\"6796\"}}\r\n{\"timestamp\":\"1475624427.399307251\",\"source\":\"guardian\",\"message\":\"guardian.api.garden-server.run.spawned\",\"log_level\":1,\"data\":{\"handle\":\"a9645f41-a32f-4fa7-58e4-98838a3191d8\",\"id\":\"337a97b6-3343-48a4-6d62-37b032f7729b\",\"session\":\"3.1.52629\",\"spec\":{\"Path\":\"/opt/resource/in\",\"Dir\":\"\",\"User\":\"root\",\"Limits\":{},\"TTY\":null}}}\r\n{\"timestamp\":\"1475624637.809005260\",\"source\":\"guardian\",\"message\":\"guardian.api.garden-server.run.exited\",\"log_level\":1,\"data\":{\"handle\":\"a9645f41-a32f-4fa7-58e4-98838a3191d8\",\"id\":\"337a97b6-3343-48a4-6d62-37b032f7729b\",\"session\":\"3.1.52629\",\"status\":0}}\r\n```\r\n\r\nin comparison to a container, which was cleaned up (same steps are executed, except that in the case of the zombiw container the reaper did not kick in):\r\n\r\n```\r\n{\"timestamp\":\"1475653406.035202265\",\"source\":\"guardian\",\"message\":\"guardian.run.exec.finished\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"id\":\"96692456-981b-454e-5180-7d841ca488a8\",\"path\":\"landscape/documentation/concourse/tasks/smoke/test.sh\",\"session\":\"9625.2\"}}\r\n{\"timestamp\":\"1475653406.035219669\",\"source\":\"guardian\",\"message\":\"guardian.run.finished\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"path\":\"landscape/documentation/concourse/tasks/smoke/test.sh\",\"session\":\"9625\"}}\r\n{\"timestamp\":\"1475653406.035238743\",\"source\":\"guardian\",\"message\":\"guardian.api.garden-server.run.spawned\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"id\":\"e25b4989-2ac0-44cf-6334-f5e893d60068\",\"session\":\"3.1.58572\",\"spec\":{\"Path\":\"landscape/documentation/concourse/tasks/smoke/test.sh\",\"Dir\r\n\":\"/tmp/build/f541ec31\",\"User\":\"root\",\"Limits\":{},\"TTY\":{}}}}\r\n{\"timestamp\":\"1475653432.606876373\",\"source\":\"guardian\",\"message\":\"guardian.api.garden-server.run.exited\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"id\":\"e25b4989-2ac0-44cf-6334-f5e893d60068\",\"session\":\"3.1.58572\",\"status\":0}}\r\n{\"timestamp\":\"1475653929.393552065\",\"source\":\"guardian\",\"message\":\"guardian.api.garden-server.reaping\",\"log_level\":1,\"data\":{\"grace-time\":\"5m0s\",\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"3.1\"}}\r\n{\"timestamp\":\"1475653929.393662930\",\"source\":\"guardian\",\"message\":\"guardian.destroy.start\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9724\"}}\r\n{\"timestamp\":\"1475653929.394035816\",\"source\":\"guardian\",\"message\":\"guardian.destroy.started\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9725\"}}\r\n{\"timestamp\":\"1475653929.394057512\",\"source\":\"guardian\",\"message\":\"guardian.destroy.state.started\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9725.1\"}}\r\n{\"timestamp\":\"1475653929.404460430\",\"source\":\"guardian\",\"message\":\"guardian.destroy.state.finished\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9725.1\"}}\r\n{\"timestamp\":\"1475653929.404501915\",\"source\":\"guardian\",\"message\":\"guardian.destroy.state\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9725\",\"state\":{\"Pid\":28253,\"Status\":\"created\"}}}\r\n{\"timestamp\":\"1475653929.404526949\",\"source\":\"guardian\",\"message\":\"guardian.destroy.delete.started\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9725.2\"}}\r\n{\"timestamp\":\"1475653929.515482426\",\"source\":\"guardian\",\"message\":\"guardian.destroy.delete.finished\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9725.2\"}}\r\n{\"timestamp\":\"1475653929.515547752\",\"source\":\"guardian\",\"message\":\"guardian.destroy.destroy.started\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9725.3\"}}\r\n{\"timestamp\":\"1475653929.517130852\",\"source\":\"guardian\",\"message\":\"guardian.destroy.destroy.finished\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9725.3\"}}\r\n{\"timestamp\":\"1475653929.517160177\",\"source\":\"guardian\",\"message\":\"guardian.destroy.finished\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9725\"}}\r\n{\"timestamp\":\"1475653929.535522699\",\"source\":\"guardian\",\"message\":\"guardian.create.containerizer-create.watch.done\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9623.1.4\"}}\r\n{\"timestamp\":\"1475653929.726925850\",\"source\":\"guardian\",\"message\":\"guardian.volume-plugin.destroying.layer-already-deleted-skipping\",\"log_level\":1,\"data\":{\"error\":\"could not find image: no such id: 76051528cd636f7704dea82da6cdd438a8e12de7cf99008ae715724c265cf0d2\",\"graphID\":\"96692456-981b-454e-5180-7d841ca488a8\",\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"id\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9728\"}}\r\n{\"timestamp\":\"1475653929.727013826\",\"source\":\"guardian\",\"message\":\"guardian.destroy.finished\",\"log_level\":1,\"data\":{\"handle\":\"96692456-981b-454e-5180-7d841ca488a8\",\"session\":\"9724\"}}\r\n```\r\n\r\n\r\nLooks like the question is why the guardian destroy\r\nhttps://github.com/cloudfoundry/garden/blob/c7ed40f0b983c8d082dcdfc3dcd5adfa1020195f/server/request_handling.go#L128\r\ndoes not kick in for the containers that do not run any processes anymore.\r\n\r\n\r\n\r\n## Steps to reproduce\r\n\r\nThe problem is hard to reproduce and the leak is slow (~2 containers per hour not being reaped)\r\n\r\n\r\n* Guardian release version garden-runc/0.8.0\r\n* Linux kernel version bosh-openstack-kvm-ubuntu-trusty-go_agent/3263.3\r\n* Concourse version 2.1.0\r\n* Go version 1.6.1 "}}
{"actor":{"display_login":"julz"}}