{"action":"created","issue":{"url":"https://api.github.com/repos/processone/ejabberd/issues/1184","repository_url":"https://api.github.com/repos/processone/ejabberd","labels_url":"https://api.github.com/repos/processone/ejabberd/issues/1184/labels{/name}","comments_url":"https://api.github.com/repos/processone/ejabberd/issues/1184/comments","events_url":"https://api.github.com/repos/processone/ejabberd/issues/1184/events","html_url":"https://github.com/processone/ejabberd/issues/1184","id":163595955,"number":1184,"title":"Old route record in mnesia's route table haven't been remove when restarting in some cases","user":{"login":"yzf","id":1593511,"avatar_url":"https://avatars.githubusercontent.com/u/1593511?v=3","gravatar_id":"","url":"https://api.github.com/users/yzf","html_url":"https://github.com/yzf","followers_url":"https://api.github.com/users/yzf/followers","following_url":"https://api.github.com/users/yzf/following{/other_user}","gists_url":"https://api.github.com/users/yzf/gists{/gist_id}","starred_url":"https://api.github.com/users/yzf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yzf/subscriptions","organizations_url":"https://api.github.com/users/yzf/orgs","repos_url":"https://api.github.com/users/yzf/repos","events_url":"https://api.github.com/users/yzf/events{/privacy}","received_events_url":"https://api.github.com/users/yzf/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/processone/ejabberd/labels/Kind:Core","name":"Kind:Core","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2016-07-04T03:08:19Z","updated_at":"2016-10-25T18:56:37Z","closed_at":null,"body":"> What version of ejabberd are you using?\r\n\r\n16.04\r\n\r\n> What operating system (version) are you using?\r\n\r\ndebian 8\r\n\r\n> How did you install ejabberd (source, package, distribution)?\r\n\r\nsource\r\n\r\n> What did not work as expected? Are there error messages in the log? What\r\n> was the unexpected behavior? What was the expected result?\r\n\r\nRecently, I study the route process of ejabberd, as we know, ejabberd routes messages according to the records in mnesia's table `route`. But I happened to find that, some old records in `route` were left behind after I restarted ejabberd, which means that the progresses whose pid is equal to these record's `pid` were not exist any more. Luckly, in ejabberd, these records happen to use the `local_hint` to route instread of sending messages to the `pid`'s progress, **this problem hasn't lead to any error**.\r\n\r\nAfter do some tests, I finally found that this problem is related to the following thins:\r\n1. Cluster\r\n2. `master` node have restarted ever\r\n\r\nIn my case, I have two nodes in my cluster, `node_1` and `node_2`. And I call `ejabberdctl join_cluster 'ejabberd@node_1'` in `node_2`. If I restart the slave node `node_2` using `ejabberdctl restart`, everything is ok, the old record of `node_2` in mnesia table route is deleted. But if I restart the master node `node_1` using `ejabberdctl restart`, the old record of `node_1` in mnesia table is left behind, after that, as `node_2` become the master node and `node_1` become the slave node, this time, restarting `node_2` will case record left behind. **It seem that the master node should not restart when there are some slave nodes.**\r\n\r\nSo, after some restarting, the records in table route:\r\n```\r\n(ejabberd@node_1)1> mnesia:dirty_read(route, <<\"xxx.xxx.xxx\">>).\r\n[{route,<<\"xxx.xxx.xxx\">>,<13073.278.0>,\r\n        {apply,ejabberd_local,route}},\r\n {route,<<\"xxx.xxx.xxx\">>,\r\n        <<\"xxx.xxx.xxx\">>,<0.320.0>,\r\n        {apply,ejabberd_local,route}},\r\n {route,<<\"xxx.xxx.xxx\">>,\r\n        <<\"xxx.xxx.xxx\">>,<13073.7790.150>,\r\n        {apply,ejabberd_local,route}},\r\n {route,<<\"xxx.xxx.xxx\">>,\r\n        <<\"xxx.xxx.xxx\">>,<0.16033.3>,\r\n        {apply,ejabberd_local,route}},\r\n {route,<<\"xxx.xxx.xxx\">>,\r\n        <<\"xxx.xxx.xxx\">>,<13073.30448.153>,\r\n        {apply,ejabberd_local,route}},\r\n {route,<<\"xxx.xxx.xxx\">>,\r\n        <<\"xxx.xxx.xxx\">>,<0.330.0>,\r\n        {apply,ejabberd_local,route}},\r\n {route,<<\"xxx.xxx.xxx\">>,\r\n        <<\"xxx.xxx.xxx\">>,<13073.327.0>,\r\n        {apply,ejabberd_local,route}},\r\n {route,<<\"xxx.xxx.xxx\">>,\r\n        <<\"xxx.xxx.xxx\">>,<0.18042.156>,\r\n        {apply,ejabberd_local,route}},\r\n {route,<<\"xxx.xxx.xxx\">>,\r\n        <<\"xxx.xxx.xxx\">>,<13073.15904.153>,\r\n        {apply,ejabberd_local,route}}]\r\n```\r\nNotice that, **only two records are valid**."},"comment":{"url":"https://api.github.com/repos/processone/ejabberd/issues/comments/256140511","html_url":"https://github.com/processone/ejabberd/issues/1184#issuecomment-256140511","issue_url":"https://api.github.com/repos/processone/ejabberd/issues/1184","id":256140511,"user":{"login":"genexp","id":816,"avatar_url":"https://avatars.githubusercontent.com/u/816?v=3","gravatar_id":"","url":"https://api.github.com/users/genexp","html_url":"https://github.com/genexp","followers_url":"https://api.github.com/users/genexp/followers","following_url":"https://api.github.com/users/genexp/following{/other_user}","gists_url":"https://api.github.com/users/genexp/gists{/gist_id}","starred_url":"https://api.github.com/users/genexp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/genexp/subscriptions","organizations_url":"https://api.github.com/users/genexp/orgs","repos_url":"https://api.github.com/users/genexp/repos","events_url":"https://api.github.com/users/genexp/events{/privacy}","received_events_url":"https://api.github.com/users/genexp/received_events","type":"User","site_admin":false},"created_at":"2016-10-25T18:56:37Z","updated_at":"2016-10-25T18:56:37Z","body":"Yeah, one thing I've noticed in production too is that it takes quite a\nwhile for a node to re-enter production depending on the number of users\nand persistent rooms.  Don't forget, stoping and starting a node requires\nmnesia to sync back up.  If you stop the other node before this happens,\nyou split brain the system.  As an experiment: Stop one, start it up, wait\na long long time, and see if the messages go away.\n\nOn Thu, Oct 20, 2016 at 10:14 AM, Fabrizio Capobianco <\nnotifications@github.com> wrote:\n\n> We use ejabberd stop, always.\n>\n> One big question: which one is the master node?? I was always under the\n> impression ejabberd is master-master, and there are no slaves. However,\n> people sometimes refer to one node as the master. And ejabberdctl has a\n> \"set_master nodename\" command, which makes me think there is actually a\n> master and some slaves... Shall we set the master to one of the cluster\n> machine, the one that is never stopped?\n>\n> —\n> You are receiving this because you commented.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/processone/ejabberd/issues/1184#issuecomment-255118044>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AAADMIn7cQibvrs7vQsoYaNyJoZzykKWks5q13dXgaJpZM4JEBPj>\n> .\n>\n"}}
{"actor":{"display_login":"genexp"}}