{"action":"created","issue":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/960","repository_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server","labels_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/960/labels{/name}","comments_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/960/comments","events_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/960/events","html_url":"https://github.com/rabbitmq/rabbitmq-server/issues/960","id":176407813,"number":960,"title":"Crash on tree handling in rabbit_variable_queue","user":{"login":"dcorbacho","id":646577,"avatar_url":"https://avatars.githubusercontent.com/u/646577?v=3","gravatar_id":"","url":"https://api.github.com/users/dcorbacho","html_url":"https://github.com/dcorbacho","followers_url":"https://api.github.com/users/dcorbacho/followers","following_url":"https://api.github.com/users/dcorbacho/following{/other_user}","gists_url":"https://api.github.com/users/dcorbacho/gists{/gist_id}","starred_url":"https://api.github.com/users/dcorbacho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcorbacho/subscriptions","organizations_url":"https://api.github.com/users/dcorbacho/orgs","repos_url":"https://api.github.com/users/dcorbacho/repos","events_url":"https://api.github.com/users/dcorbacho/events{/privacy}","received_events_url":"https://api.github.com/users/dcorbacho/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"dcorbacho","id":646577,"avatar_url":"https://avatars.githubusercontent.com/u/646577?v=3","gravatar_id":"","url":"https://api.github.com/users/dcorbacho","html_url":"https://github.com/dcorbacho","followers_url":"https://api.github.com/users/dcorbacho/followers","following_url":"https://api.github.com/users/dcorbacho/following{/other_user}","gists_url":"https://api.github.com/users/dcorbacho/gists{/gist_id}","starred_url":"https://api.github.com/users/dcorbacho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcorbacho/subscriptions","organizations_url":"https://api.github.com/users/dcorbacho/orgs","repos_url":"https://api.github.com/users/dcorbacho/repos","events_url":"https://api.github.com/users/dcorbacho/events{/privacy}","received_events_url":"https://api.github.com/users/dcorbacho/received_events","type":"User","site_admin":false},"assignees":[{"login":"dcorbacho","id":646577,"avatar_url":"https://avatars.githubusercontent.com/u/646577?v=3","gravatar_id":"","url":"https://api.github.com/users/dcorbacho","html_url":"https://github.com/dcorbacho","followers_url":"https://api.github.com/users/dcorbacho/followers","following_url":"https://api.github.com/users/dcorbacho/following{/other_user}","gists_url":"https://api.github.com/users/dcorbacho/gists{/gist_id}","starred_url":"https://api.github.com/users/dcorbacho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcorbacho/subscriptions","organizations_url":"https://api.github.com/users/dcorbacho/orgs","repos_url":"https://api.github.com/users/dcorbacho/repos","events_url":"https://api.github.com/users/dcorbacho/events{/privacy}","received_events_url":"https://api.github.com/users/dcorbacho/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-09-12T15:34:53Z","updated_at":"2016-10-26T08:54:28Z","closed_at":null,"body":"Found while testing #944, using HA queues and autoheal (same testing as for #914).\r\n\r\n```\r\n=ERROR REPORT==== 23-Aug-2016::16:05:46 ===\r\n** Generic server <0.32209.0> terminating\r\n** Last message in was {'$gen_cast',{ack,\"<E9>\",<32920.259.1>}}\r\n** When Server state == {q,\r\n                         {amqqueue,\r\n                          {resource,<<\"/\">>,queue,<<\"test_6\">>},\r\n                          false,false,none,[],<0.32209.0>,\r\n                          [<32920.60.1>],\r\n                          [<32920.60.1>],\r\n                          ['rabbit@ubuntu-c2'],\r\n                          [{vhost,<<\"/\">>},\r\n                           {name,<<\"ha-all\">>},\r\n                           {pattern,<<\".*\">>},\r\n                           {'apply-to',<<\"all\">>},\r\n                           {definition,\r\n                            [{<<\"ha-mode\">>,<<\"all\">>},\r\n                             {<<\"ha-sync-mode\">>,<<\"automatic\">>}]},\r\n                           {priority,0}],\r\n                          [{<32920.61.1>,<32920.60.1>},\r\n                           {<0.32213.0>,<0.32209.0>}],\r\n                          [],live},\r\n                         none,true,rabbit_mirror_queue_master,\r\n                         {state,\r\n                          {resource,<<\"/\">>,queue,<<\"test_6\">>},\r\n                          <0.32213.0>,<0.32212.0>,rabbit_priority_queue,\r\n                          {passthrough,rabbit_variable_queue,\r\n                           {vqstate,\r\n                            {0,{[],[]}},\r\n                            {0,{[],[]}},\r\n                            {delta,undefined,0,undefined},\r\n                            {0,{[],[]}},\r\n                            {0,{[],[]}},\r\n                            1030,\r\n                            {0,nil},\r\n                            {0,nil},\r\n                            {0,nil},\r\n                            {qistate,\r\n                             \"/tmp/rabbitmq-test-instances/rabbit/mnesia/rabbit/queues/9H1TFV3HQW3DAMFDY3SQ4VVIN\",\r\n                             {{dict,0,16,16,8,80,48,\r\n                               {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                []},\r\n                               {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                 []}}},\r\n                              []},\r\n                             undefined,0,32768,\r\n                             #Fun<rabbit_variable_queue.2.97317634>,\r\n....\r\n** Reason for termination ==\r\n** {function_clause,\r\n       [{gb_trees,delete_1,[233,nil],[{file,\"gb_trees.erl\"},{line,407}]},\r\n        {gb_trees,delete,2,[{file,\"gb_trees.erl\"},{line,403}]},\r\n        {rabbit_variable_queue,remove_pending_ack,3,\r\n            [{file,\"src/rabbit_variable_queue.erl\"},{line,1993}]},\r\n        {rabbit_variable_queue,remove_pending_ack,3,\r\n            [{file,\"src/rabbit_variable_queue.erl\"},{line,1980}]},\r\n        {rabbit_variable_queue,ack,2,\r\n            [{file,\"src/rabbit_variable_queue.erl\"},{line,643}]},\r\n        {rabbit_priority_queue,ack,2,\r\n            [{file,\"src/rabbit_priority_queue.erl\"},{line,316}]},\r\n        {rabbit_mirror_queue_master,ack,2,\r\n            [{file,\"src/rabbit_mirror_queue_master.erl\"},{line,378}]},\r\n        {rabbit_amqqueue_process,'-ack/3-fun-0-',2,\r\n            [{file,\"src/rabbit_amqqueue_process.erl\"},{line,682}]}]}\r\n```"},"comment":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/comments/256287958","html_url":"https://github.com/rabbitmq/rabbitmq-server/issues/960#issuecomment-256287958","issue_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/960","id":256287958,"user":{"login":"dcorbacho","id":646577,"avatar_url":"https://avatars.githubusercontent.com/u/646577?v=3","gravatar_id":"","url":"https://api.github.com/users/dcorbacho","html_url":"https://github.com/dcorbacho","followers_url":"https://api.github.com/users/dcorbacho/followers","following_url":"https://api.github.com/users/dcorbacho/following{/other_user}","gists_url":"https://api.github.com/users/dcorbacho/gists{/gist_id}","starred_url":"https://api.github.com/users/dcorbacho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcorbacho/subscriptions","organizations_url":"https://api.github.com/users/dcorbacho/orgs","repos_url":"https://api.github.com/users/dcorbacho/repos","events_url":"https://api.github.com/users/dcorbacho/events{/privacy}","received_events_url":"https://api.github.com/users/dcorbacho/received_events","type":"User","site_admin":false},"created_at":"2016-10-26T08:54:28Z","updated_at":"2016-10-26T08:54:28Z","body":"The root cause is not changes in master/slave status as I originally thought, but the network partition causing remote channels (in a different node) to be removed from the queue. If a message has been delivered to a remote channel and immediately after the queue process receives a `DOWN` message from the channel, the messages pending acknowledgment for this channel are requeued. Thus, later delivered to a different channel.\r\n\r\nIf the other node comes back shortly after (a few seconds on my tests), the queue might end up receiving two acknowledgments for the same ack tag: one from the channel considered `down` and one from the redelivery channel. The second ack causes the crash as the tag cannot be found."}}
{"actor":{"display_login":"dcorbacho"}}