{"action":"created","issue":{"url":"https://api.github.com/repos/chef/chef/issues/5213","repository_url":"https://api.github.com/repos/chef/chef","labels_url":"https://api.github.com/repos/chef/chef/issues/5213/labels{/name}","comments_url":"https://api.github.com/repos/chef/chef/issues/5213/comments","events_url":"https://api.github.com/repos/chef/chef/issues/5213/events","html_url":"https://github.com/chef/chef/issues/5213","id":171493711,"number":5213,"title":"yum package cache uses way too much memory","user":{"login":"johnbellone","id":241756,"avatar_url":"https://avatars.githubusercontent.com/u/241756?v=3","gravatar_id":"","url":"https://api.github.com/users/johnbellone","html_url":"https://github.com/johnbellone","followers_url":"https://api.github.com/users/johnbellone/followers","following_url":"https://api.github.com/users/johnbellone/following{/other_user}","gists_url":"https://api.github.com/users/johnbellone/gists{/gist_id}","starred_url":"https://api.github.com/users/johnbellone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johnbellone/subscriptions","organizations_url":"https://api.github.com/users/johnbellone/orgs","repos_url":"https://api.github.com/users/johnbellone/repos","events_url":"https://api.github.com/users/johnbellone/events{/privacy}","received_events_url":"https://api.github.com/users/johnbellone/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-08-16T19:09:05Z","updated_at":"2016-08-16T20:41:31Z","closed_at":null,"body":"## Description\r\n\r\nDuring a normal chef-client convergence after installing the first package using the Yum provider we experience a failure to fork when starting service due to being out of memory. After subsequent testing we found that during this time the Chef Client process was eating up more than 1.2 GB of resident memory (over 50%). \r\n\r\n## Chef Version\r\n\r\nChef Client 12.11.18\r\n\r\n## Platform Version\r\n\r\nRHEL 6.8\r\nRHEL 7.2\r\n\r\n## Replication Case\r\nThe instances that we have observed this problem on have 1 VCPU and 2 GB of memory. We are able to consistently observe this when running test-kitchen against a static Policyfile lock. This instance is connected to an internal mirror server with all of the RHEL packages on it. It is *not* connected to RH satellite.\r\n\r\nWe did some sleuthing and found that the chef-client shells out to a Python script that ships with the distribution. This script seems to iterate through the complete RPM database and write out the relevant information which is then consumed by a [regular expression](https://github.com/chef/chef/blob/master/lib/chef/provider/package/yum/yum_cache.rb) parser into the Ruby interpreter. We then took the [yum-dump.py Python script](https://github.com/chef/chef/blob/master/lib/chef/provider/package/yum/yum-dump.py) with the same arguments wrote the values to a file (~70MB). Using a simple `IO.read 'file.txt'` we found that an IRB process had ~255MB of resident memory being utilized. \r\n\r\nWhile looking through the issues we found #4122 and #4869 which may be relevant to this problem. We also found the [ruby-rpm-ffi binding gem](https://github.com/dmacvicar/ruby-rpm-ffi) which we thought may be helpful in the remediation. \r\n\r\n```bash\r\n[root@ip-1-71-67-6 ~]# wc -l yum-dump.txt \r\n49966 yum-dump.txt\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/chef/chef/issues/comments/240231463","html_url":"https://github.com/chef/chef/issues/5213#issuecomment-240231463","issue_url":"https://api.github.com/repos/chef/chef/issues/5213","id":240231463,"user":{"login":"johnbellone","id":241756,"avatar_url":"https://avatars.githubusercontent.com/u/241756?v=3","gravatar_id":"","url":"https://api.github.com/users/johnbellone","html_url":"https://github.com/johnbellone","followers_url":"https://api.github.com/users/johnbellone/followers","following_url":"https://api.github.com/users/johnbellone/following{/other_user}","gists_url":"https://api.github.com/users/johnbellone/gists{/gist_id}","starred_url":"https://api.github.com/users/johnbellone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johnbellone/subscriptions","organizations_url":"https://api.github.com/users/johnbellone/orgs","repos_url":"https://api.github.com/users/johnbellone/repos","events_url":"https://api.github.com/users/johnbellone/events{/privacy}","received_events_url":"https://api.github.com/users/johnbellone/received_events","type":"User","site_admin":false},"created_at":"2016-08-16T20:41:31Z","updated_at":"2016-08-16T20:41:31Z","body":"This seems to have been a red herring. @coderanger and I are debugging and it seems the memory issue only presents itself when passing the `source 'url'` into the `package` resource. "}}
{"actor":{"display_login":"johnbellone"}}