{"action":"created","issue":{"url":"https://api.github.com/repos/signalfx/docker-collectd-plugin/issues/35","repository_url":"https://api.github.com/repos/signalfx/docker-collectd-plugin","labels_url":"https://api.github.com/repos/signalfx/docker-collectd-plugin/issues/35/labels{/name}","comments_url":"https://api.github.com/repos/signalfx/docker-collectd-plugin/issues/35/comments","events_url":"https://api.github.com/repos/signalfx/docker-collectd-plugin/issues/35/events","html_url":"https://github.com/signalfx/docker-collectd-plugin/pull/35","id":372480442,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI0NjQ5Nzkz","number":35,"title":"Deduct the cache figure from the memory usage percentage metric","user":{"login":"chrisgilbert","id":361112,"node_id":"MDQ6VXNlcjM2MTExMg==","avatar_url":"https://avatars1.githubusercontent.com/u/361112?v=4","gravatar_id":"","url":"https://api.github.com/users/chrisgilbert","html_url":"https://github.com/chrisgilbert","followers_url":"https://api.github.com/users/chrisgilbert/followers","following_url":"https://api.github.com/users/chrisgilbert/following{/other_user}","gists_url":"https://api.github.com/users/chrisgilbert/gists{/gist_id}","starred_url":"https://api.github.com/users/chrisgilbert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrisgilbert/subscriptions","organizations_url":"https://api.github.com/users/chrisgilbert/orgs","repos_url":"https://api.github.com/users/chrisgilbert/repos","events_url":"https://api.github.com/users/chrisgilbert/events{/privacy}","received_events_url":"https://api.github.com/users/chrisgilbert/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"keitwb","id":614344,"node_id":"MDQ6VXNlcjYxNDM0NA==","avatar_url":"https://avatars2.githubusercontent.com/u/614344?v=4","gravatar_id":"","url":"https://api.github.com/users/keitwb","html_url":"https://github.com/keitwb","followers_url":"https://api.github.com/users/keitwb/followers","following_url":"https://api.github.com/users/keitwb/following{/other_user}","gists_url":"https://api.github.com/users/keitwb/gists{/gist_id}","starred_url":"https://api.github.com/users/keitwb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keitwb/subscriptions","organizations_url":"https://api.github.com/users/keitwb/orgs","repos_url":"https://api.github.com/users/keitwb/repos","events_url":"https://api.github.com/users/keitwb/events{/privacy}","received_events_url":"https://api.github.com/users/keitwb/received_events","type":"User","site_admin":false},"assignees":[{"login":"keitwb","id":614344,"node_id":"MDQ6VXNlcjYxNDM0NA==","avatar_url":"https://avatars2.githubusercontent.com/u/614344?v=4","gravatar_id":"","url":"https://api.github.com/users/keitwb","html_url":"https://github.com/keitwb","followers_url":"https://api.github.com/users/keitwb/followers","following_url":"https://api.github.com/users/keitwb/following{/other_user}","gists_url":"https://api.github.com/users/keitwb/gists{/gist_id}","starred_url":"https://api.github.com/users/keitwb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keitwb/subscriptions","organizations_url":"https://api.github.com/users/keitwb/orgs","repos_url":"https://api.github.com/users/keitwb/repos","events_url":"https://api.github.com/users/keitwb/events{/privacy}","received_events_url":"https://api.github.com/users/keitwb/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2018-10-22T11:19:29Z","updated_at":"2018-11-06T15:35:07Z","closed_at":"2018-10-30T17:26:29Z","author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/signalfx/docker-collectd-plugin/pulls/35","html_url":"https://github.com/signalfx/docker-collectd-plugin/pull/35","diff_url":"https://github.com/signalfx/docker-collectd-plugin/pull/35.diff","patch_url":"https://github.com/signalfx/docker-collectd-plugin/pull/35.patch"},"body":"Currently the calculation for memory usage of a container/cgroup being returned to SignalFX includes the Linux page cache.  This is generally considered to be incorrect, and may lead people to chase phantom memory leaks in their application.  See some other applications where they have fixed the same problem:\r\n\r\nhttps://github.com/moby/moby/issues/10824\r\nhttps://github.com/aws/amazon-ecs-agent/issues/280\r\n\r\nHere's a fix to correctly deduct the figure where it is available.\r\n\r\nFor a demonstration on why the current calculation is incorrect, you can run the following to see how I/O usage influences the overall memory usage in a cgroup:\r\n\r\n```\r\ndocker run --rm -ti alpine\r\ncat /sys/fs/cgroup/memory/memory.stat\r\ncat /sys/fs/cgroup/memory/memory.usage_in_bytes\r\ndd if=/dev/zero of=/tmp/myfile bs=1M count=100\r\ncat /sys/fs/cgroup/memory/memory.stat\r\ncat /sys/fs/cgroup/memory/memory.usage_in_bytes\r\n```\r\n\r\nYou should see that the `usage_in_bytes` value rises by 100MB just from creating a 100MB file. That file hasn't been loaded into anonymous memory by an application, but because it's now in the page cache, the container memory usage is _appearing_ to be higher.  Deducting the cache figure in `memory.stat` from the `usage_in_bytes` shows that the genuine use of anonymous memory hasn't risen.\r\n\r\nBecause the kernel can evict page cache memory, this memory should not be considered \"in use\" by the container, and it is not very useful to measure it.\r\n\r\nIf the container asks for more memory from the OS, it will get it, and page cache will be evicted where required.  Also see https://www.linuxatemyram.com for more explanations of why the page cache generally uses most, or all free memory on the machine."},"comment":{"url":"https://api.github.com/repos/signalfx/docker-collectd-plugin/issues/comments/436295656","html_url":"https://github.com/signalfx/docker-collectd-plugin/pull/35#issuecomment-436295656","issue_url":"https://api.github.com/repos/signalfx/docker-collectd-plugin/issues/35","id":436295656,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNjI5NTY1Ng==","user":{"login":"chrisgilbert","id":361112,"node_id":"MDQ6VXNlcjM2MTExMg==","avatar_url":"https://avatars1.githubusercontent.com/u/361112?v=4","gravatar_id":"","url":"https://api.github.com/users/chrisgilbert","html_url":"https://github.com/chrisgilbert","followers_url":"https://api.github.com/users/chrisgilbert/followers","following_url":"https://api.github.com/users/chrisgilbert/following{/other_user}","gists_url":"https://api.github.com/users/chrisgilbert/gists{/gist_id}","starred_url":"https://api.github.com/users/chrisgilbert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrisgilbert/subscriptions","organizations_url":"https://api.github.com/users/chrisgilbert/orgs","repos_url":"https://api.github.com/users/chrisgilbert/repos","events_url":"https://api.github.com/users/chrisgilbert/events{/privacy}","received_events_url":"https://api.github.com/users/chrisgilbert/received_events","type":"User","site_admin":false},"created_at":"2018-11-06T15:35:07Z","updated_at":"2018-11-06T15:35:07Z","author_association":"NONE","body":"We are still using the collectd based plugins at the moment.  Thanks, I'll watch the PR then and take a look when it's released 👍 "}}
{"actor":{"display_login":"chrisgilbert"}}