{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23323","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23323/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23323/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23323/events","html_url":"https://github.com/kubernetes/kubernetes/issues/23323","id":142607633,"number":23323,"title":"apiserver keeps increasing in memory until it gets oomkilled","user":{"login":"jsravn","id":522123,"avatar_url":"https://avatars.githubusercontent.com/u/522123?v=3","gravatar_id":"","url":"https://api.github.com/users/jsravn","html_url":"https://github.com/jsravn","followers_url":"https://api.github.com/users/jsravn/followers","following_url":"https://api.github.com/users/jsravn/following{/other_user}","gists_url":"https://api.github.com/users/jsravn/gists{/gist_id}","starred_url":"https://api.github.com/users/jsravn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsravn/subscriptions","organizations_url":"https://api.github.com/users/jsravn/orgs","repos_url":"https://api.github.com/users/jsravn/repos","events_url":"https://api.github.com/users/jsravn/events{/privacy}","received_events_url":"https://api.github.com/users/jsravn/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/CSI-API%20Machinery%20SIG","name":"team/CSI-API Machinery SIG","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":35,"created_at":"2016-03-22T10:30:12Z","updated_at":"2016-04-06T18:29:50Z","closed_at":null,"body":"We have a kubernetes 1.2 cluster where our apiserver keeps getting oomkilled every day or two. This is a pretty small cluster (4 nodes), and we're not sure why it only happens on this cluster. Memory usage constantly grows from less than 100MB at start to over 2GB when it will get oomkilled (the box has 4GB total ram). Restarting apiserver resets its memory usage to less than 100MB.\r\n\r\nWe have another medium cluster (12 nodes with lots of traffic) where apiserver memory usage is very stable, staying at around 700MB usage. Edit: This cluster got oomkilled as well, the memory growth was just much slower. Takes a few weeks.\r\n\r\nI've attached pprof files for both. It seems most of the space usage is in the ugorji json decoder, followed by the etcd cache.\r\n\r\n(@balooo was also looking at this with me.)\r\n\r\n[memoryleak-pprof.inuse_objects.inuse_space.pb.gz](https://github.com/kubernetes/kubernetes/files/183911/memoryleak-pprof.inuse_objects.inuse_space.pb.gz)\r\n[normal-pprof.inuse_objects.inuse_space.pb.gz](https://github.com/kubernetes/kubernetes/files/183912/normal-pprof.inuse_objects.inuse_space.pb.gz)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/206504994","html_url":"https://github.com/kubernetes/kubernetes/issues/23323#issuecomment-206504994","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/23323","id":206504994,"user":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"created_at":"2016-04-06T18:29:50Z","updated_at":"2016-04-06T18:29:50Z","body":"@wojtek-t Do we serve lists out of this cache after removing a random element?"}}