{"action":"created","issue":{"url":"https://api.github.com/repos/erlio/vernemq/issues/547","repository_url":"https://api.github.com/repos/erlio/vernemq","labels_url":"https://api.github.com/repos/erlio/vernemq/issues/547/labels{/name}","comments_url":"https://api.github.com/repos/erlio/vernemq/issues/547/comments","events_url":"https://api.github.com/repos/erlio/vernemq/issues/547/events","html_url":"https://github.com/erlio/vernemq/issues/547","id":277377180,"node_id":"MDU6SXNzdWUyNzczNzcxODA=","number":547,"title":"Subscriptions stop working after plumtree_metadata_manager crashes in cluster environment","user":{"login":"victorkendy","id":237167,"node_id":"MDQ6VXNlcjIzNzE2Nw==","avatar_url":"https://avatars3.githubusercontent.com/u/237167?v=4","gravatar_id":"","url":"https://api.github.com/users/victorkendy","html_url":"https://github.com/victorkendy","followers_url":"https://api.github.com/users/victorkendy/followers","following_url":"https://api.github.com/users/victorkendy/following{/other_user}","gists_url":"https://api.github.com/users/victorkendy/gists{/gist_id}","starred_url":"https://api.github.com/users/victorkendy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/victorkendy/subscriptions","organizations_url":"https://api.github.com/users/victorkendy/orgs","repos_url":"https://api.github.com/users/victorkendy/repos","events_url":"https://api.github.com/users/victorkendy/events{/privacy}","received_events_url":"https://api.github.com/users/victorkendy/received_events","type":"User","site_admin":false},"labels":[{"id":770160859,"node_id":"MDU6TGFiZWw3NzAxNjA4NTk=","url":"https://api.github.com/repos/erlio/vernemq/labels/need%20more%20data","name":"need more data","color":"b2ffe6","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2017-11-28T13:04:31Z","updated_at":"2018-09-21T12:47:17Z","closed_at":"2018-02-21T06:50:33Z","author_association":"NONE","body":"### Environment\r\n\r\n- VerneMQ Version: 1.2.0\r\n- OS: docker container using Ubuntu 16.04 (Xenial)\r\n- Erlang/OTP version (if building from source):\r\n\r\n### Expected behavior\r\n\r\n- Subscriptions should work after a plumtree_metadata_manager crash.\r\n\r\n### Actual behaviour\r\n\r\n- When there is an error in the plumtree_metadata_manager the vernemq server stops accepting new subscriptions even when the manager process is restarted by the supervisor.\r\n\r\nWe had a cluster with 2 vernemq instances. Everything worked fine until there was a match error inside the plumtree_metadata_manager, these are the log entries for this error:\r\n\r\n```\r\nCRASH REPORT Process <0.21109.810> with 0 neighbours exited with reason: {{{{case_clause,[]},[{plumtree_metadata_manager,store,2,[{file,\"/opt/vernemq/distdir/1.2.0/_build/default/lib/plumtree/src/plumtree_metadata_manager.erl\"},{line,472}]},{plumtree_metadata_manager,read_merge_write,2,[{file,\"/opt/vernemq/distdir/1.2.0/_build/default/lib/plumtree/src/plumtree_metadata_manager.erl\"},{line,457}]},{plumtree_metadata_manager,handle_call,3,[{file,\"/opt/vernemq/distdir/1.2.0/_build/default/lib/plumtree/src/plumtree_metadata_manager.erl\"},{line,316}]},{gen_server,try_handle_call,...},...]},...},...} in gen_fsm:terminate/7 line 559\r\n\r\nCRASH REPORT Process plumtree_metadata_manager with 0 neighbours exited with reason: no case clause matching [] in plumtree_metadata_manager:store/2 line 472 in gen_server:terminate/7 line 812\r\n\r\nCRASH REPORT Process plumtree_broadcast with 0 neighbours exited with reason: {{{case_clause,[]},[{plumtree_metadata_manager,store,2,[{file,\"/opt/vernemq/distdir/1.2.0/_build/default/lib/plumtree/src/plumtree_metadata_manager.erl\"},{line,472}]},{plumtree_metadata_manager,read_merge_write,2,[{file,\"/opt/vernemq/distdir/1.2.0/_build/default/lib/plumtree/src/plumtree_metadata_manager.erl\"},{line,457}]},{plumtree_metadata_manager,handle_call,3,[{file,\"/opt/vernemq/distdir/1.2.0/_build/default/lib/plumtree/src/plumtree_metadata_manager.erl\"},{line,316}]},{gen_server,try_handle_call,...},...]},...} in gen_server2:terminate/3 line 995\r\n\r\nCRASH REPORT Process plumtree_metadata_manager with 0 neighbours exited with reason: no case clause matching [] in plumtree_metadata_manager:store/2 line 472 in gen_server:terminate/7 line 812\r\n```\r\n\r\nAfter the error, the server continues to accept connection e publish messages, but subscriptions do not work (`router_subscriptions` metric remains constant and messages are not sent to subscribers).\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/erlio/vernemq/issues/comments/423519142","html_url":"https://github.com/erlio/vernemq/issues/547#issuecomment-423519142","issue_url":"https://api.github.com/repos/erlio/vernemq/issues/547","id":423519142,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzUxOTE0Mg==","user":{"login":"larshesel","id":679187,"node_id":"MDQ6VXNlcjY3OTE4Nw==","avatar_url":"https://avatars3.githubusercontent.com/u/679187?v=4","gravatar_id":"","url":"https://api.github.com/users/larshesel","html_url":"https://github.com/larshesel","followers_url":"https://api.github.com/users/larshesel/followers","following_url":"https://api.github.com/users/larshesel/following{/other_user}","gists_url":"https://api.github.com/users/larshesel/gists{/gist_id}","starred_url":"https://api.github.com/users/larshesel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/larshesel/subscriptions","organizations_url":"https://api.github.com/users/larshesel/orgs","repos_url":"https://api.github.com/users/larshesel/repos","events_url":"https://api.github.com/users/larshesel/events{/privacy}","received_events_url":"https://api.github.com/users/larshesel/received_events","type":"User","site_admin":false},"created_at":"2018-09-21T12:47:17Z","updated_at":"2018-09-21T12:47:17Z","author_association":"CONTRIBUTOR","body":"Hi @victorkendy I hope it's ok I revive this old issue - we've had some other cases of this problem and I just wanted to ask for a bit more information if you remember it.\r\n\r\ndid you use `allow_multiple_session=on` when you had the problem? What were the netsplit settings - the `allow_xxx_during_netsplit`?"}}
{"actor":{"display_login":"larshesel"}}