{"action":"created","issue":{"url":"https://api.github.com/repos/ben-manes/caffeine/issues/127","repository_url":"https://api.github.com/repos/ben-manes/caffeine","labels_url":"https://api.github.com/repos/ben-manes/caffeine/issues/127/labels{/name}","comments_url":"https://api.github.com/repos/ben-manes/caffeine/issues/127/comments","events_url":"https://api.github.com/repos/ben-manes/caffeine/issues/127/events","html_url":"https://github.com/ben-manes/caffeine/issues/127","id":184286733,"number":127,"title":"Cache Workers getting stuck?","user":{"login":"normanuber","id":14983089,"avatar_url":"https://avatars.githubusercontent.com/u/14983089?v=3","gravatar_id":"","url":"https://api.github.com/users/normanuber","html_url":"https://github.com/normanuber","followers_url":"https://api.github.com/users/normanuber/followers","following_url":"https://api.github.com/users/normanuber/following{/other_user}","gists_url":"https://api.github.com/users/normanuber/gists{/gist_id}","starred_url":"https://api.github.com/users/normanuber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/normanuber/subscriptions","organizations_url":"https://api.github.com/users/normanuber/orgs","repos_url":"https://api.github.com/users/normanuber/repos","events_url":"https://api.github.com/users/normanuber/events{/privacy}","received_events_url":"https://api.github.com/users/normanuber/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":35,"created_at":"2016-10-20T17:06:05Z","updated_at":"2016-10-30T20:21:06Z","closed_at":null,"body":"I saw this problem in version 2.3.1, via VisualVm, when after a few hours that a bunch of ForkJoinWorkers would be stuck on the WriteBuffer.poll() function in RUNNABLE state. Even hit an OOM when the cpu utilization became too high and my events started to get backed up. I recently upgraded to 2.3.3 and thought the problem was fixed until the below screenshots.\r\n\r\nWhen I started up the application (10/19, 7:34 am) there is one FJW in the writebuffer.poll, which is fine.\r\n<img width=\"998\" alt=\"screen shot 2016-10-19 at 7 34 41 am\" src=\"https://cloud.githubusercontent.com/assets/14983089/19569558/0a6ad1bc-96ab-11e6-9706-3505dd75cae3.png\">\r\n\r\nThis morning (10/20, 9:34am) and this is in constant RUNNABLE state. The number of workers that will be in this stuck runnable state will slowly grow over time too because I have seen this behavior happen with 2.3.1.\r\n<img width=\"993\" alt=\"screen shot 2016-10-20 at 9 34 02 am\" src=\"https://cloud.githubusercontent.com/assets/14983089/19569552/01d2bcfe-96ab-11e6-8d04-0b0936cf43a9.png\">\r\n\r\nThe cpu utilization and cpu load metrics in Grafana support an overall increase work as if the workers are busy waiting/spinlooping or something.\r\n\r\nThere are maybe 1k or so instances of the cache data structure in the application. This server has the most instances of the cache and the issue appears to manifest faster than the other servers. All of the servers have a 64GB heap.\r\n\r\nI will have to switch back to Guava considering this is a production application.\r\n\r\nThanks for your effort though!"},"comment":{"url":"https://api.github.com/repos/ben-manes/caffeine/issues/comments/257176295","html_url":"https://github.com/ben-manes/caffeine/issues/127#issuecomment-257176295","issue_url":"https://api.github.com/repos/ben-manes/caffeine/issues/127","id":257176295,"user":{"login":"ben-manes","id":378614,"avatar_url":"https://avatars.githubusercontent.com/u/378614?v=3","gravatar_id":"","url":"https://api.github.com/users/ben-manes","html_url":"https://github.com/ben-manes","followers_url":"https://api.github.com/users/ben-manes/followers","following_url":"https://api.github.com/users/ben-manes/following{/other_user}","gists_url":"https://api.github.com/users/ben-manes/gists{/gist_id}","starred_url":"https://api.github.com/users/ben-manes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ben-manes/subscriptions","organizations_url":"https://api.github.com/users/ben-manes/orgs","repos_url":"https://api.github.com/users/ben-manes/repos","events_url":"https://api.github.com/users/ben-manes/events{/privacy}","received_events_url":"https://api.github.com/users/ben-manes/received_events","type":"User","site_admin":false},"created_at":"2016-10-30T20:21:06Z","updated_at":"2016-10-30T20:21:06Z","body":"@nitsanw Can you please publish the test artifacts in future releases? Since I'm manually shading, I'd like to copy and run `MpscGrowableQueueSanityTest` against it. Rather than copy `QueueSanityTest` and its imports, simply having the test dependency would be ideal. Not sure how to query jitpack for this type of artifact."}}
{"actor":{"display_login":"ben-manes"}}