{"action":"opened","issue":{"url":"https://api.github.com/repos/cloudfoundry/garden-runc-release/issues/112","repository_url":"https://api.github.com/repos/cloudfoundry/garden-runc-release","labels_url":"https://api.github.com/repos/cloudfoundry/garden-runc-release/issues/112/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry/garden-runc-release/issues/112/comments","events_url":"https://api.github.com/repos/cloudfoundry/garden-runc-release/issues/112/events","html_url":"https://github.com/cloudfoundry/garden-runc-release/issues/112","id":390447876,"node_id":"MDU6SXNzdWUzOTA0NDc4NzY=","number":112,"title":"Containerd-backed Garden does not record out-of-memory container events when Linux OOM-killer terminates processes","user":{"login":"emalm","id":5836757,"node_id":"MDQ6VXNlcjU4MzY3NTc=","avatar_url":"https://avatars2.githubusercontent.com/u/5836757?v=4","gravatar_id":"","url":"https://api.github.com/users/emalm","html_url":"https://github.com/emalm","followers_url":"https://api.github.com/users/emalm/followers","following_url":"https://api.github.com/users/emalm/following{/other_user}","gists_url":"https://api.github.com/users/emalm/gists{/gist_id}","starred_url":"https://api.github.com/users/emalm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emalm/subscriptions","organizations_url":"https://api.github.com/users/emalm/orgs","repos_url":"https://api.github.com/users/emalm/repos","events_url":"https://api.github.com/users/emalm/events{/privacy}","received_events_url":"https://api.github.com/users/emalm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-12-12T23:07:02Z","updated_at":"2018-12-12T23:07:02Z","closed_at":null,"author_association":"MEMBER","body":"## Description\r\n\r\nWhen Garden uses containerd as its local container system instead of straight runc, it fails to report out-of-memory events in the `Events` field on the [`ContainerInfo`](https://godoc.org/code.cloudfoundry.org/garden#ContainerInfo) data about existing containers. The Diego cell rep translates these events into an `(out of memory)` annotation in the app log stream on process exit codes so that application operators and developers can ascertain that their app instance failed after the Linux kernel killed one or more of the processes inside the container.\r\n\r\n\r\n## Environment\r\n\r\n* garden-runc-release version: v1.17.1\r\n* IaaS: GCP; BOSH-Lite\r\n* Stemcell version: ubuntu-xenial/170.12\r\n* Kernel version: 4.15.0-42-generic\r\n\r\n## Steps to reproduce\r\n\r\nReproduction steps on a full CF with CF SSH enabled:\r\n\r\n1. Set `garden.experimental_containerd_mode` to `true` on the `garden` job on the Linux Diego cells (the default in cf-deployment v6.3.0 and later).\r\n1. Push a 1-instance [Dora](https://github.com/cloudfoundry/cf-acceptance-tests/tree/master/assets/dora) app with a 32MB memory limit\r\n1. Tail the Dora app logs with `cf logs dora`\r\n1. Force the container to its memory limit via `cf ssh dora -c '/home/vcap/app/stress --vm 16 --vm-bytes 2M'`\r\n1. Observe that the Dora app process exits but that no `(out of memory)` annotation is present on the `Exit status N` lines in the Dora app logs or in the exit_description in the crash event in `cf events`.\r\n\r\n## Logs\r\n\r\n`cf logs` output with containerd enabled:\r\n\r\n```\r\n   2018-12-12T13:48:48.26-0800 [SSH/0] OUT Successful remote access by 192.168.50.1:52669\r\n   2018-12-12T13:48:48.91-0800 [APP/PROC/WEB/0] OUT Exit status 137\r\n   2018-12-12T13:48:48.91-0800 [SSH/0] OUT Remote access ended for 192.168.50.1:52669\r\n   2018-12-12T13:48:48.94-0800 [CELL/SSHD/0] OUT Exit status 0\r\n   2018-12-12T13:48:54.76-0800 [CELL/0] OUT Cell f93c6cf2-2ce8-4cb6-8a35-edc80087300c stopping instance 72086376-3eb9-484f-74e9-2501\r\n   2018-12-12T13:48:54.76-0800 [CELL/0] OUT Cell f93c6cf2-2ce8-4cb6-8a35-edc80087300c destroying container for instance 72086376-3eb9-484f-74e9-2501\r\n   2018-12-12T13:48:54.96-0800 [API/0] OUT Process has crashed with type: \"web\"\r\n   2018-12-12T13:48:55.56-0800 [API/0] OUT App instance exited with guid fdb1f890-e623-4d4b-80f5-ad99ff197bfb payload: {\"instance\"=>\"72086376-3eb9-484f-74e9-2501\", \"index\"=>0, \"cell_id\"=>\"f93c6cf2-2ce8-4cb6-8a35-edc80087300c\", \"reason\"=>\"CRASHED\", \"exit_description\"=>\"APP/PROC/WEB: Exited with status 137\", \"crash_count\"=>1, \"crash_timestamp\"=>1544651334709042862, \"version\"=>\"6ec2906a-988f-4e4d-b916-2dc6af07fa6a\"}\r\n   2018-12-12T13:48:55.67-0800 [PROXY/0] OUT Exit status 137\r\n   2018-12-12T13:48:56.20-0800 [CELL/0] OUT Cell f93c6cf2-2ce8-4cb6-8a35-edc80087300c successfully destroyed container for instance 72086376-3eb9-484f-74e9-2501\r\n```\r\n\r\n`cf logs` output with the native garden-runc runner enabled:\r\n\r\n```\r\n   2018-12-12T13:24:44.71-0800 [SSH/0] OUT Successful remote access by 192.168.50.1:50287\r\n   2018-12-12T13:24:45.07-0800 [APP/PROC/WEB/0] OUT Exit status 137 (out of memory)\r\n   2018-12-12T13:24:45.07-0800 [SSH/0] OUT Remote access ended for 192.168.50.1:50287\r\n   2018-12-12T13:24:45.10-0800 [CELL/SSHD/0] OUT Exit status 0\r\n   2018-12-12T13:24:51.34-0800 [CELL/0] OUT Cell 54ac57a5-9b6f-4532-b7e4-aba023d1f649 stopping instance 9d1c8492-db22-4752-76fb-2629\r\n   2018-12-12T13:24:51.39-0800 [API/0] OUT Process has crashed with type: \"web\"\r\n   2018-12-12T13:24:51.42-0800 [API/0] OUT App instance exited with guid fdb1f890-e623-4d4b-80f5-ad99ff197bfb payload: {\"instance\"=>\"9d1c8492-db22-4752-76fb-2629\", \"index\"=>0, \"cell_id\"=>\"54ac57a5-9b6f-4532-b7e4-aba023d1f649\", \"reason\"=>\"CRASHED\", \"exit_description\"=>\"APP/PROC/WEB: Exited with status 137 (out of memory)\", \"crash_count\"=>3, \"crash_timestamp\"=>1544649891321581065, \"version\"=>\"97329d75-7617-48bd-8c91-6a44589a3f90\"}\r\n   2018-12-12T13:24:51.34-0800 [CELL/0] OUT Cell 54ac57a5-9b6f-4532-b7e4-aba023d1f649 destroying container for instance 9d1c8492-db22-4752-76fb-2629\r\n   2018-12-12T13:24:51.61-0800 [PROXY/0] OUT Exit status 137\r\n   2018-12-12T13:24:51.97-0800 [CELL/0] OUT Cell 54ac57a5-9b6f-4532-b7e4-aba023d1f649 successfully destroyed container for instance 9d1c8492-db22-4752-76fb-2629\r\n```\r\n\r\nLogs from all the BOSH jobs on the diego-cell VM with containerd enabled: [cf.diego-cell.0-20181212-150330-136274.tar.gz](https://github.com/cloudfoundry/garden-runc-release/files/2673893/cf.diego-cell.0-20181212-150330-136274.tar.gz)"}}
{"actor":{"display_login":"emalm"}}