{"action":"created","issue":{"url":"https://api.github.com/repos/apache/couchdb/issues/1571","repository_url":"https://api.github.com/repos/apache/couchdb","labels_url":"https://api.github.com/repos/apache/couchdb/issues/1571/labels{/name}","comments_url":"https://api.github.com/repos/apache/couchdb/issues/1571/comments","events_url":"https://api.github.com/repos/apache/couchdb/issues/1571/events","html_url":"https://github.com/apache/couchdb/issues/1571","id":352654780,"node_id":"MDU6SXNzdWUzNTI2NTQ3ODA=","number":1571,"title":"rexi_server dies without being restarted","user":{"login":"davisp","id":19929,"node_id":"MDQ6VXNlcjE5OTI5","avatar_url":"https://avatars3.githubusercontent.com/u/19929?v=4","gravatar_id":"","url":"https://api.github.com/users/davisp","html_url":"https://github.com/davisp","followers_url":"https://api.github.com/users/davisp/followers","following_url":"https://api.github.com/users/davisp/following{/other_user}","gists_url":"https://api.github.com/users/davisp/gists{/gist_id}","starred_url":"https://api.github.com/users/davisp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davisp/subscriptions","organizations_url":"https://api.github.com/users/davisp/orgs","repos_url":"https://api.github.com/users/davisp/repos","events_url":"https://api.github.com/users/davisp/events{/privacy}","received_events_url":"https://api.github.com/users/davisp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-08-21T18:19:02Z","updated_at":"2018-08-21T18:32:08Z","closed_at":null,"author_association":"MEMBER","body":"Turns out that rexi_server's can die in such a way that they're not restarted. This can (and has!) left a cluster without the ability to issue RPC calls effectively rendering the cluster useless.\r\n\r\nA slightly redacted log showing it happen due to hitting the process limit is:\r\n\r\n2018-08-18T21:00:05.106860Z db3.clustername <0.19934.2> -  gen_server 'rexi_server_dbcore@db1.clustername.cloudant.net' terminated with reason: system_limit at erlang:spawn_opt/1 <= erlang:spawn_monitor/3 <= rexi_server:handle_cast/2(line:71) <= gen_server:try_dispatch/4(line:593) <= gen_server:handle_msg/5(line:659) <= proc_lib:init_p_do_apply/3(line:237)#012 state: {st,6946959,7078032,{[],[]},0,0}\r\n"},"comment":{"url":"https://api.github.com/repos/apache/couchdb/issues/comments/414776879","html_url":"https://github.com/apache/couchdb/issues/1571#issuecomment-414776879","issue_url":"https://api.github.com/repos/apache/couchdb/issues/1571","id":414776879,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNDc3Njg3OQ==","user":{"login":"davisp","id":19929,"node_id":"MDQ6VXNlcjE5OTI5","avatar_url":"https://avatars3.githubusercontent.com/u/19929?v=4","gravatar_id":"","url":"https://api.github.com/users/davisp","html_url":"https://github.com/davisp","followers_url":"https://api.github.com/users/davisp/followers","following_url":"https://api.github.com/users/davisp/following{/other_user}","gists_url":"https://api.github.com/users/davisp/gists{/gist_id}","starred_url":"https://api.github.com/users/davisp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davisp/subscriptions","organizations_url":"https://api.github.com/users/davisp/orgs","repos_url":"https://api.github.com/users/davisp/repos","events_url":"https://api.github.com/users/davisp/events{/privacy}","received_events_url":"https://api.github.com/users/davisp/received_events","type":"User","site_admin":false},"created_at":"2018-08-21T18:32:08Z","updated_at":"2018-08-21T18:32:08Z","author_association":"MEMBER","body":"I was thinking more like that it should have spidered up to the top rexi supervisor rather than giving up.\r\n\r\nAlso of note was that the rexi_buffer processes were all alive. Should have mentioned that earlier as well. The half alive part of the issue I think is the bigger bug."}}
{"actor":{"display_login":"davisp"}}