{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/bugs/issues/2398","repository_url":"https://api.github.com/repos/coreos/bugs","labels_url":"https://api.github.com/repos/coreos/bugs/issues/2398/labels{/name}","comments_url":"https://api.github.com/repos/coreos/bugs/issues/2398/comments","events_url":"https://api.github.com/repos/coreos/bugs/issues/2398/events","html_url":"https://github.com/coreos/bugs/issues/2398","id":312167116,"number":2398,"title":"OOMs just writing files with k8s memory limits (upstream kernel bug)","user":{"login":"comstud","id":637328,"avatar_url":"https://avatars3.githubusercontent.com/u/637328?v=4","gravatar_id":"","url":"https://api.github.com/users/comstud","html_url":"https://github.com/comstud","followers_url":"https://api.github.com/users/comstud/followers","following_url":"https://api.github.com/users/comstud/following{/other_user}","gists_url":"https://api.github.com/users/comstud/gists{/gist_id}","starred_url":"https://api.github.com/users/comstud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/comstud/subscriptions","organizations_url":"https://api.github.com/users/comstud/orgs","repos_url":"https://api.github.com/users/comstud/repos","events_url":"https://api.github.com/users/comstud/events{/privacy}","received_events_url":"https://api.github.com/users/comstud/received_events","type":"User","site_admin":false},"labels":[{"id":418444453,"url":"https://api.github.com/repos/coreos/bugs/labels/area/stability","name":"area/stability","color":"c2e0c6","default":false},{"id":269192631,"url":"https://api.github.com/repos/coreos/bugs/labels/component/kernel","name":"component/kernel","color":"0052cc","default":false},{"id":113394315,"url":"https://api.github.com/repos/coreos/bugs/labels/kind/regression","name":"kind/regression","color":"DF0101","default":false},{"id":275179790,"url":"https://api.github.com/repos/coreos/bugs/labels/team/os","name":"team/os","color":"ffffff","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-07T03:42:21Z","updated_at":"2018-04-07T04:28:35Z","closed_at":null,"author_association":"NONE","body":"# Issue Report #\r\n\r\n<!-- Fill in either the 'Bug' or 'Feature Request' section -->\r\n\r\n## Bug ##\r\n\r\nlegacy memory cgroup proceses are OOM-killed just writing files. This is very evident when using CoreOS as a kubernetes node... configuring memory limits on pods. My upstream bug report can be found here which details the issue in a little more detail:\r\n\r\nhttps://bugzilla.kernel.org/show_bug.cgi?id=199297\r\n\r\nThe fix for this is already committed upstream and is contained in kernel version 4.16. I am reporting it here, however, due to the fact that this should affect almost anyone using CoreOS as a kubernetes node. I am not sure how far off a stable CoreOS with kernel version 4.16+ would be... so perhaps this fix should be backported to a stable coreos release?\r\n\r\nThis is the upstream commit that fixes the issue:\r\n\r\nhttps://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=1c610d5f93c7\r\n\r\n### Container Linux Version ###\r\n\r\nAll versions with linux kernel 4.11 through (but not including) version 4.16 are affected.\r\n\r\n### Environment ###\r\n\r\nWhat hardware/cloud provider/hypervisor is being used to run Container Linux?\r\nDoesn't matter.\r\n\r\n### Expected Behavior ###\r\n\r\nNo OOM kills writing files.\r\n\r\n### Actual Behavior ###\r\n\r\nprocess OOM killed writing files (or write() syscall returns ENOMEM).\r\n\r\n### Reproduction Steps ###\r\n\r\n```\r\n#!/bin/sh\r\n\r\nset -e\r\n\r\nif [ `id -u` != 0 ]; then\r\n    echo 'must run as root'\r\n    exit 1\r\nfi\r\n\r\ncg_memory_path_prefix='/sys/fs/cgroup/memory'\r\n\r\n# Make a new cgroup and remove on exit\r\ncg=`mktemp -p ${cg_memory_path_prefix} -d`\r\necho \"Created $cg\"\r\ntrap \"{ echo 0 > ${cg}/memory.force_empty; rmdir $cg; }\" EXIT\r\n\r\n# 256M\r\necho 268435456 > ${cg}/memory.limit_in_bytes\r\n\r\n# Add our pid to the cgroup\r\necho $$ > ${cg}/tasks\r\n\r\n# Create tmp file\r\ntmpfile=`mktemp -p /tmp`\r\ntrap \"rm $tmpfile\" EXIT\r\n\r\n# Write 4G\r\necho \"Writing to $tmpfile\"\r\ndd if=/dev/zero of=$tmpfile bs=1048576 count=4096\r\n# Below not reached. dd will be OOMkilled.\r\necho 'No OOM'\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/bugs/issues/comments/379431320","html_url":"https://github.com/coreos/bugs/issues/2398#issuecomment-379431320","issue_url":"https://api.github.com/repos/coreos/bugs/issues/2398","id":379431320,"user":{"login":"bgilbert","id":361374,"avatar_url":"https://avatars1.githubusercontent.com/u/361374?v=4","gravatar_id":"","url":"https://api.github.com/users/bgilbert","html_url":"https://github.com/bgilbert","followers_url":"https://api.github.com/users/bgilbert/followers","following_url":"https://api.github.com/users/bgilbert/following{/other_user}","gists_url":"https://api.github.com/users/bgilbert/gists{/gist_id}","starred_url":"https://api.github.com/users/bgilbert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgilbert/subscriptions","organizations_url":"https://api.github.com/users/bgilbert/orgs","repos_url":"https://api.github.com/users/bgilbert/repos","events_url":"https://api.github.com/users/bgilbert/events{/privacy}","received_events_url":"https://api.github.com/users/bgilbert/received_events","type":"User","site_admin":false},"created_at":"2018-04-07T04:28:35Z","updated_at":"2018-04-07T04:28:35Z","author_association":"MEMBER","body":"Thanks for the report.  That patch is included in kernels &ge; 4.14.31 and &ge; 4.15.14.  The latest Container Linux stable release, 1688.5.3, thus already has the fix, and the next releases on the alpha and beta channels will as well."}}
{"actor":{"display_login":"bgilbert"}}