{"action":"closed","number":2161,"pull_request":{"url":"https://api.github.com/repos/dCache/dcache/pulls/2161","id":58437706,"html_url":"https://github.com/dCache/dcache/pull/2161","diff_url":"https://github.com/dCache/dcache/pull/2161.diff","patch_url":"https://github.com/dCache/dcache/pull/2161.patch","issue_url":"https://api.github.com/repos/dCache/dcache/issues/2161","number":2161,"state":"closed","locked":false,"title":"(2.14) alarms: change executor to have bounded queue and discard even…","user":{"login":"alrossi","id":3985564,"avatar_url":"https://avatars.githubusercontent.com/u/3985564?v=3","gravatar_id":"","url":"https://api.github.com/users/alrossi","html_url":"https://github.com/alrossi","followers_url":"https://api.github.com/users/alrossi/followers","following_url":"https://api.github.com/users/alrossi/following{/other_user}","gists_url":"https://api.github.com/users/alrossi/gists{/gist_id}","starred_url":"https://api.github.com/users/alrossi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alrossi/subscriptions","organizations_url":"https://api.github.com/users/alrossi/orgs","repos_url":"https://api.github.com/users/alrossi/repos","events_url":"https://api.github.com/users/alrossi/events{/privacy}","received_events_url":"https://api.github.com/users/alrossi/received_events","type":"User","site_admin":false},"body":"…ts on overrun\r\n\r\nMotivation:\r\n\r\ndCache logging clients send remote events using an Asynchronous appender which delegates to a SocketAppender.\r\nThe former provides better performance on the client side by buffering events\r\nand allowing them to be consumed by the delegate.  Since the delegate in turn uses the TCP layer\r\nto send events to the server, the usual buffering and discarding of events when no connection\r\ncan be established should occur.  Hence while the network bandwith provides the upper bound on\r\nperformance when the wrapper buffer/queue fills, the client should not block in such cases.\r\n\r\nHowever, if the server becomes unresponsive (but does not close the client connection), the client can block.\r\nSeveral sites have experienced this problem when using the rdbms backend to the logging service\r\n(see RT 8879 Pool not responding: http://rt.dcache.org/Ticket/Display.html?id=8879; we have seen\r\nnearly the identical problem here on production at Fermi).  Under heavy usage or logging bursts, the\r\nexecutor queue of tasks fills up, either because of sluggish database response or connection contention,\r\nand produces an OutOfMemory error.  At this point, depending on the state of the JVM, it is possible that\r\nthe server threads hang or freeze, not allowing the domain to restart, at least immediately.  The server\r\nthus remains in an unresponsive state, holding on to the client connections, and causing the clients to\r\nblock on the threads attempting to do logging.\r\n\r\nWhile it may be possible to improve the database performance, in general there needs to be a safety valve\r\n(as there is on the billing service) whereby bursts of connection activity do not provoke an OOM.\r\nAvoiding the OOM will avoid the principal cause for the critical condition under discussion.\r\n\r\nModification:\r\n\r\nInstead of a fixed thread pool executor with unbounded blocking queue, we use a (Cached)BoundedExecutor\r\nwith queue size defined by an additional property.  When max size is reached, the service drops the event.\r\nGiven that this is not a function-critical service, we believe this compromise acceptable.  Note that we have\r\nprovided a suggested (very conservative) ratio of 128 Mb of VM heap for every 1000 queue slots, determined\r\nby testing.\r\n\r\nResult:\r\n\r\nUnder heavy load from the clients, the server discards events before the heap allocation can be used up,\r\navoiding loss of responsiveness and the blocking of clients.\r\n\r\nTarget: 2.14\r\nRequire-notes: yes\r\nRequire-book: no\r\nAcked-by: Dmitry\r\nAcked-by: Tigran","created_at":"2016-02-05T14:49:59Z","updated_at":"2016-02-08T14:32:03Z","closed_at":"2016-02-08T14:32:03Z","merged_at":"2016-02-08T14:32:03Z","merge_commit_sha":"7715545fc1f38a06cba4458bacfc39b85615d935","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/dCache/dcache/pulls/2161/commits","review_comments_url":"https://api.github.com/repos/dCache/dcache/pulls/2161/comments","review_comment_url":"https://api.github.com/repos/dCache/dcache/pulls/comments{/number}","comments_url":"https://api.github.com/repos/dCache/dcache/issues/2161/comments","statuses_url":"https://api.github.com/repos/dCache/dcache/statuses/cdee225dad336b8612e75cdabcc98b09c0fa8715","head":{"label":"alrossi:fix/2.14/alarms-bounded-exec-with-discard","ref":"fix/2.14/alarms-bounded-exec-with-discard","sha":"cdee225dad336b8612e75cdabcc98b09c0fa8715","user":{"login":"alrossi","id":3985564,"avatar_url":"https://avatars.githubusercontent.com/u/3985564?v=3","gravatar_id":"","url":"https://api.github.com/users/alrossi","html_url":"https://github.com/alrossi","followers_url":"https://api.github.com/users/alrossi/followers","following_url":"https://api.github.com/users/alrossi/following{/other_user}","gists_url":"https://api.github.com/users/alrossi/gists{/gist_id}","starred_url":"https://api.github.com/users/alrossi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alrossi/subscriptions","organizations_url":"https://api.github.com/users/alrossi/orgs","repos_url":"https://api.github.com/users/alrossi/repos","events_url":"https://api.github.com/users/alrossi/events{/privacy}","received_events_url":"https://api.github.com/users/alrossi/received_events","type":"User","site_admin":false},"repo":{"id":9675694,"name":"dcache-server","full_name":"alrossi/dcache-server","owner":{"login":"alrossi","id":3985564,"avatar_url":"https://avatars.githubusercontent.com/u/3985564?v=3","gravatar_id":"","url":"https://api.github.com/users/alrossi","html_url":"https://github.com/alrossi","followers_url":"https://api.github.com/users/alrossi/followers","following_url":"https://api.github.com/users/alrossi/following{/other_user}","gists_url":"https://api.github.com/users/alrossi/gists{/gist_id}","starred_url":"https://api.github.com/users/alrossi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alrossi/subscriptions","organizations_url":"https://api.github.com/users/alrossi/orgs","repos_url":"https://api.github.com/users/alrossi/repos","events_url":"https://api.github.com/users/alrossi/events{/privacy}","received_events_url":"https://api.github.com/users/alrossi/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/alrossi/dcache-server","description":"dCache source code","fork":true,"url":"https://api.github.com/repos/alrossi/dcache-server","forks_url":"https://api.github.com/repos/alrossi/dcache-server/forks","keys_url":"https://api.github.com/repos/alrossi/dcache-server/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alrossi/dcache-server/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alrossi/dcache-server/teams","hooks_url":"https://api.github.com/repos/alrossi/dcache-server/hooks","issue_events_url":"https://api.github.com/repos/alrossi/dcache-server/issues/events{/number}","events_url":"https://api.github.com/repos/alrossi/dcache-server/events","assignees_url":"https://api.github.com/repos/alrossi/dcache-server/assignees{/user}","branches_url":"https://api.github.com/repos/alrossi/dcache-server/branches{/branch}","tags_url":"https://api.github.com/repos/alrossi/dcache-server/tags","blobs_url":"https://api.github.com/repos/alrossi/dcache-server/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alrossi/dcache-server/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alrossi/dcache-server/git/refs{/sha}","trees_url":"https://api.github.com/repos/alrossi/dcache-server/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alrossi/dcache-server/statuses/{sha}","languages_url":"https://api.github.com/repos/alrossi/dcache-server/languages","stargazers_url":"https://api.github.com/repos/alrossi/dcache-server/stargazers","contributors_url":"https://api.github.com/repos/alrossi/dcache-server/contributors","subscribers_url":"https://api.github.com/repos/alrossi/dcache-server/subscribers","subscription_url":"https://api.github.com/repos/alrossi/dcache-server/subscription","commits_url":"https://api.github.com/repos/alrossi/dcache-server/commits{/sha}","git_commits_url":"https://api.github.com/repos/alrossi/dcache-server/git/commits{/sha}","comments_url":"https://api.github.com/repos/alrossi/dcache-server/comments{/number}","issue_comment_url":"https://api.github.com/repos/alrossi/dcache-server/issues/comments{/number}","contents_url":"https://api.github.com/repos/alrossi/dcache-server/contents/{+path}","compare_url":"https://api.github.com/repos/alrossi/dcache-server/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alrossi/dcache-server/merges","archive_url":"https://api.github.com/repos/alrossi/dcache-server/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alrossi/dcache-server/downloads","issues_url":"https://api.github.com/repos/alrossi/dcache-server/issues{/number}","pulls_url":"https://api.github.com/repos/alrossi/dcache-server/pulls{/number}","milestones_url":"https://api.github.com/repos/alrossi/dcache-server/milestones{/number}","notifications_url":"https://api.github.com/repos/alrossi/dcache-server/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alrossi/dcache-server/labels{/name}","releases_url":"https://api.github.com/repos/alrossi/dcache-server/releases{/id}","deployments_url":"https://api.github.com/repos/alrossi/dcache-server/deployments","created_at":"2013-04-25T16:02:34Z","updated_at":"2015-06-15T14:14:36Z","pushed_at":"2016-02-05T20:09:17Z","git_url":"git://github.com/alrossi/dcache-server.git","ssh_url":"git@github.com:alrossi/dcache-server.git","clone_url":"https://github.com/alrossi/dcache-server.git","svn_url":"https://github.com/alrossi/dcache-server","homepage":null,"size":343877,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"dCache:2.14","ref":"2.14","sha":"1cd6877277b800191093d194e80d45d4c147c2d1","user":{"login":"dCache","id":1384620,"avatar_url":"https://avatars.githubusercontent.com/u/1384620?v=3","gravatar_id":"","url":"https://api.github.com/users/dCache","html_url":"https://github.com/dCache","followers_url":"https://api.github.com/users/dCache/followers","following_url":"https://api.github.com/users/dCache/following{/other_user}","gists_url":"https://api.github.com/users/dCache/gists{/gist_id}","starred_url":"https://api.github.com/users/dCache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dCache/subscriptions","organizations_url":"https://api.github.com/users/dCache/orgs","repos_url":"https://api.github.com/users/dCache/repos","events_url":"https://api.github.com/users/dCache/events{/privacy}","received_events_url":"https://api.github.com/users/dCache/received_events","type":"Organization","site_admin":false},"repo":{"id":9113580,"name":"dcache","full_name":"dCache/dcache","owner":{"login":"dCache","id":1384620,"avatar_url":"https://avatars.githubusercontent.com/u/1384620?v=3","gravatar_id":"","url":"https://api.github.com/users/dCache","html_url":"https://github.com/dCache","followers_url":"https://api.github.com/users/dCache/followers","following_url":"https://api.github.com/users/dCache/following{/other_user}","gists_url":"https://api.github.com/users/dCache/gists{/gist_id}","starred_url":"https://api.github.com/users/dCache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dCache/subscriptions","organizations_url":"https://api.github.com/users/dCache/orgs","repos_url":"https://api.github.com/users/dCache/repos","events_url":"https://api.github.com/users/dCache/events{/privacy}","received_events_url":"https://api.github.com/users/dCache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/dCache/dcache","description":"dCache - a system for storing and retrieving huge amounts of data, distributed among a large number of heterogenous server nodes, under a single virtual filesystem tree with a variety of standard access methods","fork":false,"url":"https://api.github.com/repos/dCache/dcache","forks_url":"https://api.github.com/repos/dCache/dcache/forks","keys_url":"https://api.github.com/repos/dCache/dcache/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dCache/dcache/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dCache/dcache/teams","hooks_url":"https://api.github.com/repos/dCache/dcache/hooks","issue_events_url":"https://api.github.com/repos/dCache/dcache/issues/events{/number}","events_url":"https://api.github.com/repos/dCache/dcache/events","assignees_url":"https://api.github.com/repos/dCache/dcache/assignees{/user}","branches_url":"https://api.github.com/repos/dCache/dcache/branches{/branch}","tags_url":"https://api.github.com/repos/dCache/dcache/tags","blobs_url":"https://api.github.com/repos/dCache/dcache/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dCache/dcache/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dCache/dcache/git/refs{/sha}","trees_url":"https://api.github.com/repos/dCache/dcache/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dCache/dcache/statuses/{sha}","languages_url":"https://api.github.com/repos/dCache/dcache/languages","stargazers_url":"https://api.github.com/repos/dCache/dcache/stargazers","contributors_url":"https://api.github.com/repos/dCache/dcache/contributors","subscribers_url":"https://api.github.com/repos/dCache/dcache/subscribers","subscription_url":"https://api.github.com/repos/dCache/dcache/subscription","commits_url":"https://api.github.com/repos/dCache/dcache/commits{/sha}","git_commits_url":"https://api.github.com/repos/dCache/dcache/git/commits{/sha}","comments_url":"https://api.github.com/repos/dCache/dcache/comments{/number}","issue_comment_url":"https://api.github.com/repos/dCache/dcache/issues/comments{/number}","contents_url":"https://api.github.com/repos/dCache/dcache/contents/{+path}","compare_url":"https://api.github.com/repos/dCache/dcache/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dCache/dcache/merges","archive_url":"https://api.github.com/repos/dCache/dcache/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dCache/dcache/downloads","issues_url":"https://api.github.com/repos/dCache/dcache/issues{/number}","pulls_url":"https://api.github.com/repos/dCache/dcache/pulls{/number}","milestones_url":"https://api.github.com/repos/dCache/dcache/milestones{/number}","notifications_url":"https://api.github.com/repos/dCache/dcache/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dCache/dcache/labels{/name}","releases_url":"https://api.github.com/repos/dCache/dcache/releases{/id}","deployments_url":"https://api.github.com/repos/dCache/dcache/deployments","created_at":"2013-03-30T08:58:49Z","updated_at":"2016-02-07T10:04:24Z","pushed_at":"2016-02-08T14:32:03Z","git_url":"git://github.com/dCache/dcache.git","ssh_url":"git@github.com:dCache/dcache.git","clone_url":"https://github.com/dCache/dcache.git","svn_url":"https://github.com/dCache/dcache","homepage":"http://www.dcache.org","size":345550,"stargazers_count":30,"watchers_count":30,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":28,"mirror_url":null,"open_issues_count":38,"forks":28,"open_issues":38,"watchers":30,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/dCache/dcache/pulls/2161"},"html":{"href":"https://github.com/dCache/dcache/pull/2161"},"issue":{"href":"https://api.github.com/repos/dCache/dcache/issues/2161"},"comments":{"href":"https://api.github.com/repos/dCache/dcache/issues/2161/comments"},"review_comments":{"href":"https://api.github.com/repos/dCache/dcache/pulls/2161/comments"},"review_comment":{"href":"https://api.github.com/repos/dCache/dcache/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/dCache/dcache/pulls/2161/commits"},"statuses":{"href":"https://api.github.com/repos/dCache/dcache/statuses/cdee225dad336b8612e75cdabcc98b09c0fa8715"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"jstarek","id":1275700,"avatar_url":"https://avatars.githubusercontent.com/u/1275700?v=3","gravatar_id":"","url":"https://api.github.com/users/jstarek","html_url":"https://github.com/jstarek","followers_url":"https://api.github.com/users/jstarek/followers","following_url":"https://api.github.com/users/jstarek/following{/other_user}","gists_url":"https://api.github.com/users/jstarek/gists{/gist_id}","starred_url":"https://api.github.com/users/jstarek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jstarek/subscriptions","organizations_url":"https://api.github.com/users/jstarek/orgs","repos_url":"https://api.github.com/users/jstarek/repos","events_url":"https://api.github.com/users/jstarek/events{/privacy}","received_events_url":"https://api.github.com/users/jstarek/received_events","type":"User","site_admin":false},"comments":0,"review_comments":0,"commits":1,"additions":27,"deletions":8,"changed_files":5}}