{"action":"created","issue":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/953","repository_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server","labels_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/953/labels{/name}","comments_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/953/comments","events_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/953/events","html_url":"https://github.com/rabbitmq/rabbitmq-server/issues/953","id":176333140,"number":953,"title":"Amqqueue record deleted during partial partitions using HA","user":{"login":"dcorbacho","id":646577,"avatar_url":"https://avatars.githubusercontent.com/u/646577?v=3","gravatar_id":"","url":"https://api.github.com/users/dcorbacho","html_url":"https://github.com/dcorbacho","followers_url":"https://api.github.com/users/dcorbacho/followers","following_url":"https://api.github.com/users/dcorbacho/following{/other_user}","gists_url":"https://api.github.com/users/dcorbacho/gists{/gist_id}","starred_url":"https://api.github.com/users/dcorbacho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcorbacho/subscriptions","organizations_url":"https://api.github.com/users/dcorbacho/orgs","repos_url":"https://api.github.com/users/dcorbacho/repos","events_url":"https://api.github.com/users/dcorbacho/events{/privacy}","received_events_url":"https://api.github.com/users/dcorbacho/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"dcorbacho","id":646577,"avatar_url":"https://avatars.githubusercontent.com/u/646577?v=3","gravatar_id":"","url":"https://api.github.com/users/dcorbacho","html_url":"https://github.com/dcorbacho","followers_url":"https://api.github.com/users/dcorbacho/followers","following_url":"https://api.github.com/users/dcorbacho/following{/other_user}","gists_url":"https://api.github.com/users/dcorbacho/gists{/gist_id}","starred_url":"https://api.github.com/users/dcorbacho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcorbacho/subscriptions","organizations_url":"https://api.github.com/users/dcorbacho/orgs","repos_url":"https://api.github.com/users/dcorbacho/repos","events_url":"https://api.github.com/users/dcorbacho/events{/privacy}","received_events_url":"https://api.github.com/users/dcorbacho/received_events","type":"User","site_admin":false},"assignees":[{"login":"dcorbacho","id":646577,"avatar_url":"https://avatars.githubusercontent.com/u/646577?v=3","gravatar_id":"","url":"https://api.github.com/users/dcorbacho","html_url":"https://github.com/dcorbacho","followers_url":"https://api.github.com/users/dcorbacho/followers","following_url":"https://api.github.com/users/dcorbacho/following{/other_user}","gists_url":"https://api.github.com/users/dcorbacho/gists{/gist_id}","starred_url":"https://api.github.com/users/dcorbacho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcorbacho/subscriptions","organizations_url":"https://api.github.com/users/dcorbacho/orgs","repos_url":"https://api.github.com/users/dcorbacho/repos","events_url":"https://api.github.com/users/dcorbacho/events{/privacy}","received_events_url":"https://api.github.com/users/dcorbacho/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2016-09-12T09:37:25Z","updated_at":"2016-09-15T07:33:52Z","closed_at":null,"body":"All of the following errors seem the consequence of the same problem: a rabbit amqqueue record is deleted while testing partial partitions with HA and autoheal, during the testing of #944.\r\n\r\n`{error, not_found}` seems to be caused when in another node a master just finished synchronisation, stops with reason `normal` and is re-started as a slave. This termination of the master processes causes the record to be deleted, thus other nodes cannot access to it. Why the master stops? Still investigating it. Whatever happens, the termination at this point shouldn't delete the record (it has 2 ways of termination, should use the other).\r\n```\r\n** Reason for termination ==\r\n** {{badmatch,{error,not_found}},\r\n    [{rabbit_mirror_queue_master,stop_all_slaves,2,\r\n                                 [{file,\"src/rabbit_mirror_queue_master.erl\"},\r\n                                  {line,206}]},\r\n     {rabbit_mirror_queue_master,delete_and_terminate,2,\r\n                                 [{file,\"src/rabbit_mirror_queue_master.erl\"},\r\n                                  {line,202}]},\r\n     {rabbit_amqqueue_process,'-terminate_delete/3-fun-1-',6,\r\n                              [{file,\"src/rabbit_amqqueue_process.erl\"},\r\n                               {line,283}]},\r\n     {rabbit_amqqueue_process,terminate_shutdown,2,\r\n                              [{file,\"src/rabbit_amqqueue_process.erl\"},\r\n                               {line,308}]},\r\n     {gen_server2,terminate,3,[{file,\"src/gen_server2.erl\"},{line,1129}]},\r\n     {proc_lib,wake_up,3,[{file,\"proc_lib.erl\"},{line,257}]}]}\r\n```\r\n```\r\n** Reason for termination ==\r\n** {{badmatch,{error,not_found}},\r\n    [{rabbit_amqqueue_process,i,2,\r\n                              [{file,\"src/rabbit_amqqueue_process.erl\"},\r\n                               {line,904}]},\r\n     {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,\r\n                              [{file,\"src/rabbit_amqqueue_process.erl\"},\r\n                               {line,859}]},\r\n     {rabbit_amqqueue_process,'-infos/2-lc$^0/1-0-',2,\r\n                              [{file,\"src/rabbit_amqqueue_process.erl\"},\r\n                               {line,859}]},\r\n     {rabbit_amqqueue_process,emit_stats,2,\r\n                              [{file,\"src/rabbit_amqqueue_process.erl\"},\r\n                               {line,940}]},\r\n     {rabbit_event,if_enabled,3,[{file,\"src/rabbit_event.erl\"},{line,137}]},\r\n     {rabbit_amqqueue_process,handle_pre_hibernate,1,\r\n                              [{file,\"src/rabbit_amqqueue_process.erl\"},\r\n                               {line,1374}]},\r\n     {gen_server2,pre_hibernate,1,[{file,\"src/gen_server2.erl\"},{line,704}]},\r\n     {proc_lib,wake_up,3,[{file,\"proc_lib.erl\"},{line,257}]}]}\r\n```\r\n\r\n```\r\n=SUPERVISOR REPORT==== 24-Aug-2016::09:51:55 ===\r\n     Supervisor: {<0.1695.0>,rabbit_amqqueue_sup}\r\n     Context:    start_error\r\n     Reason:     {{badmatch,{error,not_found}},\r\n                  [{rabbit_prequeue,init,2,\r\n                                    [{file,\"src/rabbit_prequeue.erl\"},\r\n                                     {line,62}]},\r\n                   {gen_server2,init_it,6,\r\n                                [{file,\"src/gen_server2.erl\"},{line,544}]},\r\n                   {proc_lib,init_p_do_apply,3,\r\n                             [{file,\"proc_lib.erl\"},{line,247}]}]}\r\n     Offender:   [{pid,{restarting,<0.21965.0>}},\r\n                  {name,rabbit_amqqueue},\r\n                  {mfargs,\r\n                      {rabbit_prequeue,start_link,\r\n                          [{amqqueue,\r\n                               {resource,<<\"/\">>,queue,<<\"test_22\">>},\r\n                               false,false,none,[],<32923.1555.0>,[],[],[],\r\n                               [{vhost,<<\"/\">>},\r\n                                {name,<<\"ha-all\">>},\r\n                                {pattern,<<\".*\">>},\r\n                                {'apply-to',<<\"all\">>},\r\n                                {definition,\r\n                                    [{<<\"ha-mode\">>,<<\"all\">>},\r\n                                     {<<\"ha-sync-mode\">>,<<\"automatic\">>}]},\r\n                                {priority,0}],\r\n                               [{<32923.1558.0>,<32923.1555.0>}],\r\n                               [],live},\r\n                           slave,<0.1694.0>]}},\r\n                  {restart_type,intrinsic},\r\n                  {shutdown,30000},\r\n                  {child_type,worker}]\r\n```\r\n```\r\n** Reason for termination ==\r\n** {{badmatch,{error,not_found}},\r\n    [{rabbit_mirror_queue_master,terminate,2,\r\n                                 [{file,\"src/rabbit_mirror_queue_master.erl\"},\r\n                                  {line,186}]},\r\n     {rabbit_amqqueue_process,terminate_shutdown,2,\r\n                              [{file,\"src/rabbit_amqqueue_process.erl\"},\r\n                               {line,308}]},\r\n     {gen_server2,terminate,3,[{file,\"src/gen_server2.erl\"},{line,1129}]},\r\n     {proc_lib,wake_up,3,[{file,\"proc_lib.erl\"},{line,257}]}]}\r\n** In 'terminate' callback with reason ==\r\n** shutdown\r\n```\r\n```\r\n=SUPERVISOR REPORT==== 31-Aug-2016::13:01:41 ===\r\n     Supervisor: {<0.383.0>,rabbit_amqqueue_sup}\r\n     Context:    child_terminated\r\n     Reason:     {{badmatch,{error,not_found}},\r\n                  [{rabbit_amqqueue_process,handle_cast,2,\r\n                       [{file,\"src/rabbit_amqqueue_process.erl\"},{line,1288}]},\r\n                   {gen_server2,handle_msg,2,\r\n                       [{file,\"src/gen_server2.erl\"},{line,1032}]},\r\n                   {proc_lib,init_p_do_apply,3,\r\n                       [{file,\"proc_lib.erl\"},{line,247}]}]}\r\n```\r\n```\r\n=CRASH REPORT==== 31-Aug-2016::13:02:59 ===\r\n  crasher:\r\n    initial call: rabbit_prequeue:init/1\r\n    pid: <0.1832.0>\r\n    registered_name: []\r\n    exception exit: {{badmatch,{error,not_found}},\r\n                     [{rabbit_mirror_queue_master,sync_mirrors,3,\r\n                          [{file,\"src/rabbit_mirror_queue_master.erl\"},\r\n                           {line,151}]},\r\n                      {rabbit_amqqueue_process,handle_call,3,\r\n                          [{file,\"src/rabbit_amqqueue_process.erl\"},\r\n                           {line,1148}]},\r\n                      {gen_server2,handle_msg,2,\r\n                          [{file,\"src/gen_server2.erl\"},{line,1011}]},\r\n                      {proc_lib,wake_up,3,\r\n                          [{file,\"proc_lib.erl\"},{line,257}]}]}\r\n      in function  gen_server2:terminate/3 (src/gen_server2.erl, line 1143)\r\n    ancestors: [<0.1831.0>,rabbit_amqqueue_sup_sup,rabbit_sup,<0.1575.0>]\r\n```"},"comment":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/comments/247257616","html_url":"https://github.com/rabbitmq/rabbitmq-server/issues/953#issuecomment-247257616","issue_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/953","id":247257616,"user":{"login":"dcorbacho","id":646577,"avatar_url":"https://avatars.githubusercontent.com/u/646577?v=3","gravatar_id":"","url":"https://api.github.com/users/dcorbacho","html_url":"https://github.com/dcorbacho","followers_url":"https://api.github.com/users/dcorbacho/followers","following_url":"https://api.github.com/users/dcorbacho/following{/other_user}","gists_url":"https://api.github.com/users/dcorbacho/gists{/gist_id}","starred_url":"https://api.github.com/users/dcorbacho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcorbacho/subscriptions","organizations_url":"https://api.github.com/users/dcorbacho/orgs","repos_url":"https://api.github.com/users/dcorbacho/repos","events_url":"https://api.github.com/users/dcorbacho/events{/privacy}","received_events_url":"https://api.github.com/users/dcorbacho/received_events","type":"User","site_admin":false},"created_at":"2016-09-15T07:33:52Z","updated_at":"2016-09-15T07:33:52Z","body":"Another case where the queue is deleted is if the GM detects membership lost (as [here](https://github.com/rabbitmq/rabbitmq-server/blob/master/src/gm.erl#L758)), so it terminates with reason `normal` causing the master to terminate in such way. Thus, deleting the record."}}
{"actor":{"display_login":"dcorbacho"}}