{"action":"created","issue":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/216","repository_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server","labels_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/216/labels{/name}","comments_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/216/comments","events_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/216/events","html_url":"https://github.com/rabbitmq/rabbitmq-server/issues/216","id":93505436,"number":216,"title":"Dead lettering more than two times results in a crashed queue","user":{"login":"riyad","id":99679,"avatar_url":"https://avatars.githubusercontent.com/u/99679?v=3","gravatar_id":"","url":"https://api.github.com/users/riyad","html_url":"https://github.com/riyad","followers_url":"https://api.github.com/users/riyad/followers","following_url":"https://api.github.com/users/riyad/following{/other_user}","gists_url":"https://api.github.com/users/riyad/gists{/gist_id}","starred_url":"https://api.github.com/users/riyad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/riyad/subscriptions","organizations_url":"https://api.github.com/users/riyad/orgs","repos_url":"https://api.github.com/users/riyad/repos","events_url":"https://api.github.com/users/riyad/events{/privacy}","received_events_url":"https://api.github.com/users/riyad/received_events","type":"User","site_admin":false},"labels":[{"id":178461384,"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/labels/bug","name":"bug","color":"fc2929","default":true},{"id":178461425,"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/labels/effort-low","name":"effort-low","color":"88b3e4","default":false}],"state":"closed","locked":false,"assignee":{"login":"michaelklishin","id":1090,"avatar_url":"https://avatars.githubusercontent.com/u/1090?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelklishin","html_url":"https://github.com/michaelklishin","followers_url":"https://api.github.com/users/michaelklishin/followers","following_url":"https://api.github.com/users/michaelklishin/following{/other_user}","gists_url":"https://api.github.com/users/michaelklishin/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelklishin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelklishin/subscriptions","organizations_url":"https://api.github.com/users/michaelklishin/orgs","repos_url":"https://api.github.com/users/michaelklishin/repos","events_url":"https://api.github.com/users/michaelklishin/events{/privacy}","received_events_url":"https://api.github.com/users/michaelklishin/received_events","type":"User","site_admin":false},"assignees":[{"login":"michaelklishin","id":1090,"avatar_url":"https://avatars.githubusercontent.com/u/1090?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelklishin","html_url":"https://github.com/michaelklishin","followers_url":"https://api.github.com/users/michaelklishin/followers","following_url":"https://api.github.com/users/michaelklishin/following{/other_user}","gists_url":"https://api.github.com/users/michaelklishin/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelklishin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelklishin/subscriptions","organizations_url":"https://api.github.com/users/michaelklishin/orgs","repos_url":"https://api.github.com/users/michaelklishin/repos","events_url":"https://api.github.com/users/michaelklishin/events{/privacy}","received_events_url":"https://api.github.com/users/michaelklishin/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/milestones/6","html_url":"https://github.com/rabbitmq/rabbitmq-server/milestones/3.5.4","labels_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/milestones/6/labels","id":1125868,"number":6,"title":"3.5.4","description":null,"creator":{"login":"dumbbell","id":159804,"avatar_url":"https://avatars.githubusercontent.com/u/159804?v=3","gravatar_id":"","url":"https://api.github.com/users/dumbbell","html_url":"https://github.com/dumbbell","followers_url":"https://api.github.com/users/dumbbell/followers","following_url":"https://api.github.com/users/dumbbell/following{/other_user}","gists_url":"https://api.github.com/users/dumbbell/gists{/gist_id}","starred_url":"https://api.github.com/users/dumbbell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dumbbell/subscriptions","organizations_url":"https://api.github.com/users/dumbbell/orgs","repos_url":"https://api.github.com/users/dumbbell/repos","events_url":"https://api.github.com/users/dumbbell/events{/privacy}","received_events_url":"https://api.github.com/users/dumbbell/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":23,"state":"closed","created_at":"2015-05-22T15:10:24Z","updated_at":"2015-09-11T08:48:13Z","due_on":"2015-06-17T12:00:00Z","closed_at":"2015-07-22T15:03:18Z"},"comments":38,"created_at":"2015-07-07T11:07:43Z","updated_at":"2016-11-14T17:11:23Z","closed_at":"2015-07-08T11:53:02Z","body":"We have a similar setup to #161 where we publish messages into (unique) timeout queues that drop the messages back into their original queues for retrying.\n\nOur setup started to loose messages after we updated to RabbitMQ 3.5.3. Looking at the logs we found this:\n\n```\n=ERROR REPORT==== 23-Jun-2015::11:47:19 ===\n** Generic server <0.678.0> terminating\n** Last message in was {drop_expired,1}\n** When Server state == \n[...]\n** Reason for termination ==\n** {{case_clause,{value,{<<\"count\">>,signedint,1}}},\n    [{rabbit_dead_letter,x_death_event_key,3,[]},\n     {rabbit_dead_letter,ensure_xdeath_event_count,2,[]},\n     {rabbit_dead_letter,ensure_xdeath_event_count,2,[]},\n     {rabbit_dead_letter,'-group_by_queue_and_reason/1-fun-0-',3,[]},\n     {lists,foldl,3,[{file,\"lists.erl\"},{line,1197}]},\n     {rabbit_dead_letter,group_by_queue_and_reason,1,[]},\n     {rabbit_dead_letter,update_x_death_header,2,[]},\n     {rabbit_dead_letter,'-make_msg/5-fun-2-',8,[]}]}\n\n=ERROR REPORT==== 23-Jun-2015::11:47:19 ===\nRestarting crashed queue 'toq-gen2-REDACTED_QUEUE_NAME-96aba311-624f-4dc0-802b-70138917816c' in vhost '/'.\n```\n\nWe were able to reproduce this everytime a message was to drop out of the timeout queue (toq) for the third time (hence the \"gen2\" part of the queue name in the logs).\n\nFull log (names of queues and exchanges redacted):\n\n```\n=ERROR REPORT==== 23-Jun-2015::11:47:19 ===\n** Generic server <0.678.0> terminating\n** Last message in was {drop_expired,1}\n** When Server state == {q,\n                         {amqqueue,\n                          {resource,<<\"/\">>,queue,\n                           <<\"toq-gen2-REDACTED_QUEUE_NAME-96aba311-624f-4dc0-802b-70138917816c\">>},\n                          false,false,none,\n                          [{<<\"x-message-ttl\">>,signedint,9000},\n                           {<<\"x-dead-letter-exchange\">>,longstr,\n                            <<\"REDACTED_EXCHANGE.dlx\">>},\n                           {<<\"x-expires\">>,signedint,10000},\n                           {<<\"x-dead-letter-routing-key\">>,longstr,\n                            <<\"REDACTED_QUEUE_NAME\">>}],\n                          <0.678.0>,[],[],[],undefined,[],[],live},\n                         none,false,rabbit_priority_queue,\n                         {passthrough,rabbit_variable_queue,\n                          {vqstate,\n                           {0,{[],[]}},\n                           {0,{[],[]}},\n                           {delta,undefined,0,undefined},\n                           {0,{[],[]}},\n                           {1,\n                            {[],\n                             [{msg_status,0,\n                               <<156,89,98,190,170,144,189,166,217,107,135,227,\n                                 40,191,240,59>>,\n                               {basic_message,\n                                {resource,<<\"/\">>,exchange,<<>>},\n                                [<<\"toq-gen2-REDACTED_QUEUE_NAME-96aba311-624f-4dc0-802b-70138917816c\">>],\n                                {content,60,\n                                 {'P_basic',undefined,undefined,\n                                  [{<<\"x-message-id\">>,longstr,\n                                    <<\"96aba311-624f-4dc0-802b-70138917816c\">>},\n                                   {<<\"x-puka-delivery-tag\">>,signedint,4},\n                                   {<<\"x-death\">>,array,\n                                    [{table,\n                                      [{<<\"count\">>,signedint,1},\n                                       {<<\"exchange\">>,longstr,<<>>},\n                                       {<<\"time\">>,timestamp,1435052830},\n                                       {<<\"queue\">>,longstr,\n                                        <<\"toq-gen1-REDACTED_QUEUE_NAME-96aba311-624f-4dc0-802b-70138917816c\">>},\n                                       {<<\"reason\">>,longstr,<<\"expired\">>},\n                                       {<<\"routing-keys\">>,array,\n                                        [{longstr,\n                                          <<\"toq-gen1-REDACTED_QUEUE_NAME-96aba311-624f-4dc0-802b-70138917816c\">>}]}]},\n                                     {table,\n                                      [{<<\"count\">>,signedint,1},\n                                       {<<\"exchange\">>,longstr,<<>>},\n                                       {<<\"routing-keys\">>,array,\n                                        [{longstr,\n                                          <<\"toq-gen0-REDACTED_QUEUE_NAME-96aba311-624f-4dc0-802b-70138917816c\">>}]},\n                                       {<<\"queue\">>,longstr,\n                                        <<\"toq-gen0-REDACTED_QUEUE_NAME-96aba311-624f-4dc0-802b-70138917816c\">>},\n                                       {<<\"reason\">>,longstr,<<\"expired\">>},\n                                       {<<\"time\">>,timestamp,1435052826}]}]},\n                                   {<<\"x-origin-queue\">>,longstr,\n                                    <<\"REDACTED_QUEUE_NAME\">>}],\n                                  undefined,undefined,undefined,undefined,\n                                  undefined,undefined,undefined,undefined,\n                                  undefined,undefined,undefined},\n                                 <<32,0,0,0,3,242,12,120,45,109,101,115,115,97,\n                                   103,101,45,105,100,83,0,0,0,36,57,54,97,98,\n                                   97,51,49,49,45,54,50,52,102,45,52,100,99,48,\n                                   45,56,48,50,98,45,55,48,49,51,56,57,49,55,\n                                   56,49,54,99,19,120,45,112,117,107,97,45,100,\n                                   101,108,105,118,101,114,121,45,116,97,103,\n                                   73,0,0,0,4,7,120,45,100,101,97,116,104,65,0,\n                                   0,3,26,70,0,0,1,136,5,99,111,117,110,116,73,\n                                   0,0,0,1,8,101,120,99,104,97,110,103,101,83,\n                                   0,0,0,0,4,116,105,109,101,84,0,0,0,0,85,137,\n                                   43,30,5,113,117,101,117,101,83,0,0,0,150,\n                                   116,111,113,45,103,101,110,49,45,95,116,101,\n                                   115,116,95,112,97,121,109,101,110,116,95,\n                                   114,101,102,117,110,100,95,111,110,95,102,\n                                   105,110,97,108,95,100,101,97,116,104,46,112,\n                                   121,58,58,116,101,115,116,95,112,117,98,108,\n                                   105,115,104,95,111,110,95,102,105,110,97,\n                                   108,95,100,101,97,116,104,95,48,48,52,57,48,\n                                   101,54,53,45,49,57,98,49,45,52,55,52,53,45,\n                                   98,56,49,100,45,99,53,57,100,49,52,52,56,53,\n                                   57,54,101,45,57,54,97,98,97,51,49,49,45,54,\n                                   50,52,102,45,52,100,99,48,45,56,48,50,98,45,\n                                   55,48,49,51,56,57,49,55,56,49,54,99,6,114,\n                                   101,97,115,111,110,83,0,0,0,7,101,120,112,\n                                   105,114,101,100,12,114,111,117,116,105,110,\n                                   103,45,107,101,121,115,65,0,0,0,155,83,0,0,\n                                   0,150,116,111,113,45,103,101,110,49,45,95,\n                                   116,101,115,116,95,112,97,121,109,101,110,\n                                   116,95,114,101,102,117,110,100,95,111,110,\n                                   95,102,105,110,97,108,95,100,101,97,116,104,\n                                   46,112,121,58,58,116,101,115,116,95,112,117,\n                                   98,108,105,115,104,95,111,110,95,102,105,\n                                   110,97,108,95,100,101,97,116,104,95,48,48,\n                                   52,57,48,101,54,53,45,49,57,98,49,45,52,55,\n                                   52,53,45,98,56,49,100,45,99,53,57,100,49,52,\n                                   52,56,53,57,54,101,45,57,54,97,98,97,51,49,\n                                   49,45,54,50,52,102,45,52,100,99,48,45,56,48,\n                                   50,98,45,55,48,49,51,56,57,49,55,56,49,54,\n                                   99,70,0,0,1,136,5,99,111,117,110,116,73,0,0,\n                                   0,1,8,101,120,99,104,97,110,103,101,83,0,0,\n                                   0,0,12,114,111,117,116,105,110,103,45,107,\n                                   101,121,115,65,0,0,0,155,83,0,0,0,150,116,\n                                   111,113,45,103,101,110,48,45,95,116,101,115,\n                                   116,95,112,97,121,109,101,110,116,95,114,\n                                   101,102,117,110,100,95,111,110,95,102,105,\n                                   110,97,108,95,100,101,97,116,104,46,112,121,\n                                   58,58,116,101,115,116,95,112,117,98,108,105,\n                                   115,104,95,111,110,95,102,105,110,97,108,95,\n                                   100,101,97,116,104,95,48,48,52,57,48,101,54,\n                                   53,45,49,57,98,49,45,52,55,52,53,45,98,56,\n                                   49,100,45,99,53,57,100,49,52,52,56,53,57,54,\n                                   101,45,57,54,97,98,97,51,49,49,45,54,50,52,\n                                   102,45,52,100,99,48,45,56,48,50,98,45,55,48,\n                                   49,51,56,57,49,55,56,49,54,99,5,113,117,101,\n                                   117,101,83,0,0,0,150,116,111,113,45,103,101,\n                                   110,48,45,95,116,101,115,116,95,112,97,121,\n                                   109,101,110,116,95,114,101,102,117,110,100,\n                                   95,111,110,95,102,105,110,97,108,95,100,101,\n                                   97,116,104,46,112,121,58,58,116,101,115,116,\n                                   95,112,117,98,108,105,115,104,95,111,110,95,\n                                   102,105,110,97,108,95,100,101,97,116,104,95,\n                                   48,48,52,57,48,101,54,53,45,49,57,98,49,45,\n                                   52,55,52,53,45,98,56,49,100,45,99,53,57,100,\n                                   49,52,52,56,53,57,54,101,45,57,54,97,98,97,\n                                   51,49,49,45,54,50,52,102,45,52,100,99,48,45,\n                                   56,48,50,98,45,55,48,49,51,56,57,49,55,56,\n                                   49,54,99,6,114,101,97,115,111,110,83,0,0,0,\n                                   7,101,120,112,105,114,101,100,4,116,105,109,\n                                   101,84,0,0,0,0,85,137,43,26,14,120,45,111,\n                                   114,105,103,105,110,45,113,117,101,117,101,\n                                   83,0,0,0,104,95,116,101,115,116,95,112,97,\n                                   121,109,101,110,116,95,114,101,102,117,110,\n                                   100,95,111,110,95,102,105,110,97,108,95,100,\n                                   101,97,116,104,46,112,121,58,58,116,101,115,\n                                   116,95,112,117,98,108,105,115,104,95,111,\n                                   110,95,102,105,110,97,108,95,100,101,97,116,\n                                   104,95,48,48,52,57,48,101,54,53,45,49,57,98,\n                                   49,45,52,55,52,53,45,98,56,49,100,45,99,53,\n                                   57,100,49,52,52,56,53,57,54,101>>,\n                                 rabbit_framing_amqp_0_9_1,\n                                 [<<\"REDACTED_MESSAGE_BODY\">>]},\n                                <<156,89,98,190,170,144,189,166,217,107,135,\n                                  227,40,191,240,59>>,\n                                false},\n                               false,false,false,false,queue_index,\n                               {message_properties,1435052839418192,false,\n                                55}}]}},\n                           1,\n                           {0,nil},\n                           {0,nil},\n                           {0,nil},\n                           {qistate,\n                            \"/var/lib/rabbitmq/mnesia/REDACTED_NODE_NAME/queues/BB5UQYHDZNVEPMDAZY48D2OBL\",\n                            {{dict,0,16,16,8,80,48,\n                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],\n                               []},\n                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],\n                                []}}},\n                             []},\n                            undefined,0,65536,\n                            #Fun<rabbit_variable_queue.2.117761292>,\n                            #Fun<rabbit_variable_queue.3.48316793>,\n                            {0,nil},\n                            {0,nil}},\n                           {undefined,\n                            {client_msstate,msg_store_transient,\n                             <<185,208,160,140,133,214,105,177,135,249,236,77,\n                               148,196,41,229>>,\n                             {dict,0,16,16,8,80,48,\n                              {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],\n                               []},\n                              {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],\n                                []}}},\n                             {state,286792,\n                              \"/var/lib/rabbitmq/mnesia/REDACTED_NODE_NAME/msg_store_transient\"},\n                             rabbit_msg_store_ets_index,\n                             \"/var/lib/rabbitmq/mnesia/REDACTED_NODE_NAME/msg_store_transient\",\n                             <0.393.0>,290889,282695,294986,299083}},\n                           false,0,1,55,0,0,0,infinity,1,0,0,55,0,0,\n                           {rates,0.1294337302079736,0.0,0.0,0.0,\n                            {1435,52831,419020}},\n                           {0,nil},\n                           {0,nil},\n                           {0,nil},\n                           {0,nil},\n                           0,0,0,0}},\n                         {state,\n                          {queue,[],[],0},\n                          {inactive,1435052830418220,9010,1.0}},\n                         10000,undefined,undefined,\n                         {erlang,#Ref<0.0.0.3748>},\n                         {state,fine,5000,undefined},\n                         {0,nil},\n                         9000,\n                         {erlang,#Ref<0.0.0.3756>},\n                         1435052839418192,\n                         {state,\n                          {dict,1,16,16,8,80,48,\n                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},\n                           {{[],\n                             [[<0.639.0>|#Ref<0.0.0.3755>]],\n                             [],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},\n                          delegate},\n                         {resource,<<\"/\">>,exchange,<<\"REDACTED_EXCHANGE.dlx\">>},\n                         <<\"REDACTED_QUEUE_NAME\">>,\n                         undefined,undefined,1,running}\n** Reason for termination ==\n** {{case_clause,{value,{<<\"count\">>,signedint,1}}},\n    [{rabbit_dead_letter,x_death_event_key,3,[]},\n     {rabbit_dead_letter,ensure_xdeath_event_count,2,[]},\n     {rabbit_dead_letter,ensure_xdeath_event_count,2,[]},\n     {rabbit_dead_letter,'-group_by_queue_and_reason/1-fun-0-',3,[]},\n     {lists,foldl,3,[{file,\"lists.erl\"},{line,1197}]},\n     {rabbit_dead_letter,group_by_queue_and_reason,1,[]},\n     {rabbit_dead_letter,update_x_death_header,2,[]},\n     {rabbit_dead_letter,'-make_msg/5-fun-2-',8,[]}]}\n\n=ERROR REPORT==== 23-Jun-2015::11:47:19 ===\nRestarting crashed queue 'toq-gen2-REDACTED_QUEUE_NAME-96aba311-624f-4dc0-802b-70138917816c' in vhost '/'.\n```\n"},"comment":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/comments/260397286","html_url":"https://github.com/rabbitmq/rabbitmq-server/issues/216#issuecomment-260397286","issue_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/216","id":260397286,"user":{"login":"michaelklishin","id":1090,"avatar_url":"https://avatars.githubusercontent.com/u/1090?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelklishin","html_url":"https://github.com/michaelklishin","followers_url":"https://api.github.com/users/michaelklishin/followers","following_url":"https://api.github.com/users/michaelklishin/following{/other_user}","gists_url":"https://api.github.com/users/michaelklishin/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelklishin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelklishin/subscriptions","organizations_url":"https://api.github.com/users/michaelklishin/orgs","repos_url":"https://api.github.com/users/michaelklishin/repos","events_url":"https://api.github.com/users/michaelklishin/events{/privacy}","received_events_url":"https://api.github.com/users/michaelklishin/received_events","type":"User","site_admin":false},"created_at":"2016-11-14T17:11:22Z","updated_at":"2016-11-14T17:11:22Z","body":"The `x-death` value is an array of objects (in JSON terms), so we need to compare what your app is sending. It is certainly possible for apps to confuse RabbitMQ by sending invalid modified headers, we have seen this several times before."}}
{"actor":{"display_login":"michaelklishin"}}