{"action":"opened","issue":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/790","repository_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server","labels_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/790/labels{/name}","comments_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/790/comments","events_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/790/events","html_url":"https://github.com/rabbitmq/rabbitmq-server/issues/790","id":154293575,"number":790,"title":"RabbitMQ Corruption?","user":{"login":"lusid","id":203284,"avatar_url":"https://avatars.githubusercontent.com/u/203284?v=3","gravatar_id":"","url":"https://api.github.com/users/lusid","html_url":"https://github.com/lusid","followers_url":"https://api.github.com/users/lusid/followers","following_url":"https://api.github.com/users/lusid/following{/other_user}","gists_url":"https://api.github.com/users/lusid/gists{/gist_id}","starred_url":"https://api.github.com/users/lusid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lusid/subscriptions","organizations_url":"https://api.github.com/users/lusid/orgs","repos_url":"https://api.github.com/users/lusid/repos","events_url":"https://api.github.com/users/lusid/events{/privacy}","received_events_url":"https://api.github.com/users/lusid/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-05-11T17:02:32Z","updated_at":"2016-05-11T17:02:32Z","closed_at":null,"body":"Been running RabbitMQ for a long time with no issues. Today, it crashed and refuses to restart with the current mnesia directory intact. This is the log from boot up. At this point, I would like to know if there's anything I can do to correct the issue and keep it from happening again in the future. Thanks.\r\n\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:31 ===\r\nStarting RabbitMQ 3.6.1 on Erlang 18.3\r\nCopyright (C) 2007-2016 Pivotal Software, Inc.\r\nLicensed under the MPL.  See http://www.rabbitmq.com/\r\n\r\n              RabbitMQ 3.6.1. Copyright (C) 2007-2016 Pivotal Software, Inc.\r\n  ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/\r\n  ##  ##\r\n  ##########  Logs: tty\r\n  ######  ##        tty\r\n  ##########\r\n              Starting broker...\r\n=INFO REPORT==== 11-May-2016::16:47:31 ===\r\nnode           : rabbit@rabbit1\r\nhome dir       : /var/lib/rabbitmq\r\nconfig file(s) : /etc/rabbitmq/rabbitmq.config\r\ncookie hash    : S3kjhR411k0Mjsj48nQOpQ==\r\nlog            : tty\r\nsasl log       : tty\r\ndatabase dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbit1\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:33 ===\r\nMemory limit set to 41900MB of 104751MB total.\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:33 ===\r\nDisk free limit set to 50MB\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:33 ===\r\nLimiting to approx 524188 file handles (471767 sockets)\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:33 ===\r\nFHC read buffering:  OFF\r\nFHC write buffering: ON\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:33 ===\r\nPriority queues enabled, real BQ is rabbit_variable_queue\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\nManagement plugin: using rates mode 'basic'\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\nmsg_store_transient: using rabbit_msg_store_ets_index to provide index\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\nmsg_store_persistent: using rabbit_msg_store_ets_index to provide index\r\n\r\n=WARNING REPORT==== 11-May-2016::16:47:34 ===\r\nmsg_store_persistent: rebuilding indices from scratch\r\n\r\n=ERROR REPORT==== 11-May-2016::16:47:34 ===\r\n** Generic server <0.190.0> terminating\r\n** Last message in was {'$gen_cast',\r\n                           {submit_async,\r\n                               #Fun<rabbit_queue_index.32.122150458>}}\r\n** When Server state == undefined\r\n** Reason for termination == \r\n** {{case_clause,undefined},\r\n    [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                         [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n     {rabbit_queue_index,parse_segment_entries,3,\r\n                         [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n     {rabbit_queue_index,segment_entries_foldr,3,\r\n                         [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n     {rabbit_queue_index,scan_segments,3,\r\n                         [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n     {rabbit_queue_index,queue_index_walker_reader,2,\r\n                         [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n     {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                         [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n     {worker_pool_worker,handle_cast,2,\r\n                         [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n     {gen_server2,handle_msg,2,[{file,\"src/gen_server2.erl\"},{line,1049}]}]}\r\n\r\n=CRASH REPORT==== 11-May-2016::16:47:34 ===\r\n  crasher:\r\n    initial call: gen:init_it/6\r\n    pid: <0.190.0>\r\n    registered_name: []\r\n    exception exit: {{case_clause,undefined},\r\n                     [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                          [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                      {rabbit_queue_index,parse_segment_entries,3,\r\n                          [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                      {rabbit_queue_index,segment_entries_foldr,3,\r\n                          [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                      {rabbit_queue_index,scan_segments,3,\r\n                          [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                      {rabbit_queue_index,queue_index_walker_reader,2,\r\n                          [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                      {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                          [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                      {worker_pool_worker,handle_cast,2,\r\n                          [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                      {gen_server2,handle_msg,2,\r\n                          [{file,\"src/gen_server2.erl\"},{line,1049}]}]}\r\n      in function  gen_server2:terminate/3 (src/gen_server2.erl, line 1160)\r\n    ancestors: [worker_pool_sup,rabbit_sup,<0.169.0>]\r\n    messages: []\r\n    links: [<0.186.0>,<0.253.0>,#Port<0.23224>]\r\n    dictionary: [{{#Ref<0.0.1.92>,fhc_handle},\r\n                   {handle,{file_descriptor,prim_file,{#Port<0.23224>,59}},\r\n                           #Ref<0.0.1.92>,0,false,0,infinity,[],<<>>,0,0,0,0,\r\n                           0,false,\r\n                           \"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/55PQL71G7QHWBXJ1890G3IGB6/journal.jif\",\r\n                           [write,binary,raw,read],\r\n                           [{write_buffer,infinity}],\r\n                           true,true,-576460747686627800}},\r\n                  {worker_pool_name,worker_pool},\r\n                  {{\"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/55PQL71G7QHWBXJ1890G3IGB6/journal.jif\",\r\n                    fhc_file},\r\n                   {file,1,true}},\r\n                  {worker_pool_worker,true},\r\n                  {fhc_age_tree,{1,\r\n                                 {{-576460747686627800,#Ref<0.0.1.92>},\r\n                                  true,nil,nil}}}]\r\n    trap_exit: false\r\n    status: running\r\n    heap_size: 2586\r\n    stack_size: 27\r\n    reductions: 9065\r\n  neighbours:\r\n    neighbour: [{pid,<0.189.0>},\r\n                  {registered_name,[]},\r\n                  {initial_call,\r\n                      {gen,init_it,\r\n                          ['Argument__1','Argument__2','Argument__3',\r\n                           'Argument__4','Argument__5','Argument__6']}},\r\n                  {current_function,\r\n                      {rabbit_queue_index,parse_segment_entries,3}},\r\n                  {ancestors,[worker_pool_sup,rabbit_sup,<0.169.0>]},\r\n                  {messages,[]},\r\n                  {links,[<0.186.0>,<0.253.0>,#Port<0.23104>]},\r\n                  {dictionary,\r\n                      [{worker_pool_name,worker_pool},\r\n                       {{#Ref<0.0.1.37>,fhc_handle},\r\n                        {handle,\r\n                            {file_descriptor,prim_file,{#Port<0.23104>,47}},\r\n                            #Ref<0.0.1.37>,0,false,0,infinity,[],<<>>,0,0,0,0,\r\n                            0,false,\r\n                            \"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/25SJZ9XC23UPXPIKAKPDBCS5W/journal.jif\",\r\n                            [write,binary,raw,read],\r\n                            [{write_buffer,infinity}],\r\n                            true,true,-576460747689849873}},\r\n                       {worker_pool_worker,true},\r\n                       {{\"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/25SJZ9XC23UPXPIKAKPDBCS5W/journal.jif\",\r\n                         fhc_file},\r\n                        {file,1,true}},\r\n                       {fhc_age_tree,\r\n                           {1,\r\n                            {{-576460747689849873,#Ref<0.0.1.37>},\r\n                             true,nil,nil}}}]},\r\n                  {trap_exit,false},\r\n                  {status,runnable},\r\n                  {heap_size,17731},\r\n                  {stack_size,146},\r\n                  {reductions,15532}]\r\n    neighbour: [{pid,<0.191.0>},\r\n                  {registered_name,[]},\r\n                  {initial_call,\r\n                      {gen,init_it,\r\n                          ['Argument__1','Argument__2','Argument__3',\r\n                           'Argument__4','Argument__5','Argument__6']}},\r\n                  {current_function,\r\n                      {rabbit_queue_index,parse_segment_entries,3}},\r\n                  {ancestors,[worker_pool_sup,rabbit_sup,<0.169.0>]},\r\n                  {messages,[]},\r\n                  {links,[<0.186.0>,<0.253.0>,#Port<0.23115>]},\r\n                  {dictionary,\r\n                      [{worker_pool_name,worker_pool},\r\n                       {worker_pool_worker,true},\r\n                       {fhc_age_tree,\r\n                           {1,\r\n                            {{-576460747689177448,#Ref<0.0.1.48>},\r\n                             true,nil,nil}}},\r\n                       {{#Ref<0.0.1.48>,fhc_handle},\r\n                        {handle,\r\n                            {file_descriptor,prim_file,{#Port<0.23115>,51}},\r\n                            #Ref<0.0.1.48>,0,false,0,infinity,[],<<>>,0,0,0,0,\r\n                            0,false,\r\n                            \"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/7MAI08GJLSXCOUYETL865OQN9/journal.jif\",\r\n                            [write,binary,raw,read],\r\n                            [{write_buffer,infinity}],\r\n                            true,true,-576460747689177448}},\r\n                       {{\"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/7MAI08GJLSXCOUYETL865OQN9/journal.jif\",\r\n                         fhc_file},\r\n                        {file,1,true}}]},\r\n                  {trap_exit,false},\r\n                  {status,runnable},\r\n                  {heap_size,28690},\r\n                  {stack_size,188},\r\n                  {reductions,18195}]\r\n    neighbour: [{pid,<0.193.0>},\r\n                  {registered_name,[]},\r\n                  {initial_call,\r\n                      {gen,init_it,\r\n                          ['Argument__1','Argument__2','Argument__3',\r\n                           'Argument__4','Argument__5','Argument__6']}},\r\n                  {current_function,{prim_file,drv_get_response,1}},\r\n                  {ancestors,[worker_pool_sup,rabbit_sup,<0.169.0>]},\r\n                  {messages,[]},\r\n                  {links,[<0.186.0>,<0.253.0>,#Port<0.23220>,#Port<0.23248>]},\r\n                  {dictionary,\r\n                      [{worker_pool_name,worker_pool},\r\n                       {{#Ref<0.0.1.89>,fhc_handle},\r\n                        {handle,\r\n                            {file_descriptor,prim_file,{#Port<0.23220>,48}},\r\n                            #Ref<0.0.1.89>,0,false,0,infinity,[],<<>>,0,0,0,0,\r\n                            0,false,\r\n                            \"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/61YYRDC0RX5V1PARMH43V9OAL/journal.jif\",\r\n                            [write,binary,raw,read],\r\n                            [{write_buffer,infinity}],\r\n                            true,true,-576460747687039277}},\r\n                       {{#Ref<0.0.1.102>,fhc_handle},\r\n                        {handle,\r\n                            {file_descriptor,prim_file,{#Port<0.23248>,45}},\r\n                            #Ref<0.0.1.102>,0,false,0,0,[],<<>>,0,0,0,0,0,\r\n                            false,\r\n                            \"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/61YYRDC0RX5V1PARMH43V9OAL/0.idx\",\r\n                            [binary,raw,read],\r\n                            [],false,true,-576460747686136586}},\r\n                       {worker_pool_worker,true},\r\n                       {fhc_age_tree,\r\n                           {2,\r\n                            {{-576460747687039277,#Ref<0.0.1.89>},\r\n                             true,nil,\r\n                             {{-576460747686136586,#Ref<0.0.1.102>},\r\n                              true,nil,nil}}}},\r\n                       {{\"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/61YYRDC0RX5V1PARMH43V9OAL/journal.jif\",\r\n                         fhc_file},\r\n                        {file,1,true}},\r\n                       {{\"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/61YYRDC0RX5V1PARMH43V9OAL/0.idx\",\r\n                         fhc_file},\r\n                        {file,1,false}}]},\r\n                  {trap_exit,false},\r\n                  {status,waiting},\r\n                  {heap_size,4185},\r\n                  {stack_size,50},\r\n                  {reductions,7381}]\r\n    neighbour: [{pid,<0.202.0>},\r\n                  {registered_name,[]},\r\n                  {initial_call,\r\n                      {gen,init_it,\r\n                          ['Argument__1','Argument__2','Argument__3',\r\n                           'Argument__4','Argument__5','Argument__6']}},\r\n                  {current_function,{prim_file,drv_get_response,1}},\r\n                  {ancestors,[worker_pool_sup,rabbit_sup,<0.169.0>]},\r\n                  {messages,[]},\r\n                  {links,[<0.186.0>,<0.253.0>,#Port<0.23174>,#Port<0.23251>]},\r\n                  {dictionary,\r\n                      [{worker_pool_name,worker_pool},\r\n                       {{#Ref<0.0.2.1157>,fhc_handle},\r\n                        {handle,\r\n                            {file_descriptor,prim_file,{#Port<0.23174>,55}},\r\n                            #Ref<0.0.2.1157>,0,false,0,infinity,[],<<>>,0,0,0,\r\n                            0,0,false,\r\n                            \"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/1E74NS2VNRQ3HF5N79O34K7MG/journal.jif\",\r\n                            [write,binary,raw,read],\r\n                            [{write_buffer,infinity}],\r\n                            true,true,-576460747688280151}},\r\n                       {worker_pool_worker,true},\r\n                       {{\"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/1E74NS2VNRQ3HF5N79O34K7MG/journal.jif\",\r\n                         fhc_file},\r\n                        {file,1,true}},\r\n                       {fhc_age_tree,\r\n                           {1,\r\n                            {{-576460747688280151,#Ref<0.0.2.1157>},\r\n                             true,nil,nil}}}]},\r\n                  {trap_exit,false},\r\n                  {status,waiting},\r\n                  {heap_size,987},\r\n                  {stack_size,46},\r\n                  {reductions,6872}]\r\n    neighbour: [{pid,<0.200.0>},\r\n                  {registered_name,[]},\r\n                  {initial_call,\r\n                      {gen,init_it,\r\n                          ['Argument__1','Argument__2','Argument__3',\r\n                           'Argument__4','Argument__5','Argument__6']}},\r\n                  {current_function,{gen,do_call,4}},\r\n                  {ancestors,[worker_pool_sup,rabbit_sup,<0.169.0>]},\r\n                  {messages,[]},\r\n                  {links,[<0.186.0>,<0.253.0>,#Port<0.23199>]},\r\n                  {dictionary,\r\n                      [{worker_pool_name,worker_pool},\r\n                       {worker_pool_worker,true},\r\n                       {fhc_age_tree,\r\n                           {1,\r\n                            {{-576460747687531248,#Ref<0.0.12.42>},\r\n                             true,nil,nil}}},\r\n                       {{\"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/7NH6QC18ZUMKIC2IX89K9MO1L/journal.jif\",\r\n                         fhc_file},\r\n                        {file,1,true}},\r\n                       {{#Ref<0.0.12.42>,fhc_handle},\r\n                        {handle,\r\n                            {file_descriptor,prim_file,{#Port<0.23199>,49}},\r\n                            #Ref<0.0.12.42>,0,false,0,infinity,[],<<>>,0,0,0,\r\n                            0,0,false,\r\n                            \"/var/lib/rabbitmq/mnesia/rabbit@rabbit1/queues/7NH6QC18ZUMKIC2IX89K9MO1L/journal.jif\",\r\n                            [write,binary,raw,read],\r\n                            [{write_buffer,infinity}],\r\n                            true,true,-576460747687531248}}]},\r\n                  {trap_exit,false},\r\n                  {status,waiting},\r\n                  {heap_size,6772},\r\n                  {stack_size,59},\r\n                  {reductions,26230}]\r\n    neighbour: [{pid,<0.253.0>},\r\n                  {registered_name,[]},\r\n                  {initial_call,\r\n                      {gen,init_it,\r\n                          ['Argument__1','Argument__2','Argument__3',\r\n                           'Argument__4','Argument__5','Argument__6']}},\r\n                  {current_function,{gen_server2,process_next_msg,1}},\r\n                  {ancestors,[msg_store_persistent,rabbit_sup,<0.169.0>]},\r\n                  {messages,\r\n                      [{'$gen_call',\r\n                           {<0.200.0>,#Ref<0.0.2.1176>},\r\n                           {in,{<<51,247,183,68,140,61,142,210,212,58,4,23,190,\r\n                                  174,141,233>>,\r\n                                1}}}]},\r\n                  {links,\r\n                      [<0.200.0>,<0.202.0>,<0.249.0>,<0.190.0>,<0.193.0>,\r\n                       <0.191.0>,<0.189.0>]},\r\n                  {dictionary,[]},\r\n                  {trap_exit,false},\r\n                  {status,runnable},\r\n                  {heap_size,1598},\r\n                  {stack_size,7},\r\n                  {reductions,1208}]\r\n\r\n=SUPERVISOR REPORT==== 11-May-2016::16:47:34 ===\r\n     Supervisor: {local,worker_pool_sup}\r\n     Context:    child_terminated\r\n     Reason:     {{case_clause,undefined},\r\n                  [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                   {rabbit_queue_index,parse_segment_entries,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                   {rabbit_queue_index,segment_entries_foldr,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                   {rabbit_queue_index,scan_segments,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                   {rabbit_queue_index,queue_index_walker_reader,2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                   {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                   {worker_pool_worker,handle_cast,2,\r\n                       [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                   {gen_server2,handle_msg,2,\r\n                       [{file,\"src/gen_server2.erl\"},{line,1049}]}]}\r\n     Offender:   [{pid,<0.190.0>},\r\n                  {id,3},\r\n                  {mfargs,{worker_pool_worker,start_link,[worker_pool]}},\r\n                  {restart_type,transient},\r\n                  {shutdown,4294967295},\r\n                  {child_type,worker}]\r\n\r\n\r\n=CRASH REPORT==== 11-May-2016::16:47:34 ===\r\n  crasher:\r\n    initial call: gen:init_it/7\r\n    pid: <0.249.0>\r\n    registered_name: []\r\n    exception exit: {{{case_clause,undefined},\r\n                      [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                           [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                       {rabbit_queue_index,parse_segment_entries,3,\r\n                           [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                       {rabbit_queue_index,segment_entries_foldr,3,\r\n                           [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                       {rabbit_queue_index,scan_segments,3,\r\n                           [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                       {rabbit_queue_index,queue_index_walker_reader,2,\r\n                           [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                       {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                           [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                       {worker_pool_worker,handle_cast,2,\r\n                           [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                       {gen_server2,handle_msg,2,\r\n                           [{file,\"src/gen_server2.erl\"},{line,1049}]}]},\r\n                     {gen_server2,call,[<0.253.0>,out,infinity]}}\r\n      in function  gen_server2:init_it/6 (src/gen_server2.erl, line 601)\r\n    ancestors: [rabbit_sup,<0.169.0>]\r\n    messages: [{'EXIT',<0.253.0>,\r\n                      {{case_clause,undefined},\r\n                       [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                            [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                        {rabbit_queue_index,parse_segment_entries,3,\r\n                            [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                        {rabbit_queue_index,segment_entries_foldr,3,\r\n                            [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                        {rabbit_queue_index,scan_segments,3,\r\n                            [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                        {rabbit_queue_index,queue_index_walker_reader,2,\r\n                            [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                        {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                            [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                        {worker_pool_worker,handle_cast,2,\r\n                            [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                        {gen_server2,handle_msg,2,\r\n                            [{file,\"src/gen_server2.erl\"},{line,1049}]}]}}]\r\n    links: [<0.252.0>,<0.170.0>]\r\n    dictionary: []\r\n    trap_exit: true\r\n    status: running\r\n    heap_size: 4185\r\n    stack_size: 27\r\n    reductions: 7056\r\n  neighbours:\r\n    neighbour: [{pid,<0.252.0>},\r\n                  {registered_name,[]},\r\n                  {initial_call,{gen,init_it,\r\n                                     ['Argument__1','Argument__2',\r\n                                      'Argument__3','Argument__4',\r\n                                      'Argument__5','Argument__6']}},\r\n                  {current_function,{gen_server2,process_next_msg,1}},\r\n                  {ancestors,[msg_store_persistent,rabbit_sup,<0.169.0>]},\r\n                  {messages,[]},\r\n                  {links,[<0.249.0>]},\r\n                  {dictionary,[]},\r\n                  {trap_exit,false},\r\n                  {status,waiting},\r\n                  {heap_size,376},\r\n                  {stack_size,7},\r\n                  {reductions,85}]\r\n\r\n=SUPERVISOR REPORT==== 11-May-2016::16:47:34 ===\r\n     Supervisor: {local,worker_pool_sup}\r\n     Context:    child_terminated\r\n     Reason:     {{case_clause,undefined},\r\n                  [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                   {rabbit_queue_index,parse_segment_entries,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                   {rabbit_queue_index,segment_entries_foldr,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                   {rabbit_queue_index,scan_segments,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                   {rabbit_queue_index,queue_index_walker_reader,2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                   {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                   {worker_pool_worker,handle_cast,2,\r\n                       [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                   {gen_server2,handle_msg,2,\r\n                       [{file,\"src/gen_server2.erl\"},{line,1049}]}]}\r\n     Offender:   [{pid,<0.189.0>},\r\n                  {id,2},\r\n                  {mfargs,{worker_pool_worker,start_link,[worker_pool]}},\r\n                  {restart_type,transient},\r\n                  {shutdown,4294967295},\r\n                  {child_type,worker}]\r\n\r\n\r\n=SUPERVISOR REPORT==== 11-May-2016::16:47:34 ===\r\n     Supervisor: {local,worker_pool_sup}\r\n     Context:    child_terminated\r\n     Reason:     {{case_clause,undefined},\r\n                  [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                   {rabbit_queue_index,parse_segment_entries,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                   {rabbit_queue_index,segment_entries_foldr,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                   {rabbit_queue_index,scan_segments,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                   {rabbit_queue_index,queue_index_walker_reader,2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                   {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                   {worker_pool_worker,handle_cast,2,\r\n                       [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                   {gen_server2,handle_msg,2,\r\n                       [{file,\"src/gen_server2.erl\"},{line,1049}]}]}\r\n     Offender:   [{pid,<0.191.0>},\r\n                  {id,4},\r\n                  {mfargs,{worker_pool_worker,start_link,[worker_pool]}},\r\n                  {restart_type,transient},\r\n                  {shutdown,4294967295},\r\n                  {child_type,worker}]\r\n\r\n\r\n=CRASH REPORT==== 11-May-2016::16:47:34 ===\r\n  crasher:\r\n    initial call: application_master:init/4\r\n    pid: <0.168.0>\r\n    registered_name: []\r\n    exception exit: {bad_return,\r\n                     {{rabbit,start,[normal,[]]},\r\n                      {'EXIT',\r\n                       {{badmatch,\r\n                         {error,\r\n                          {{{{case_clause,undefined},\r\n                             [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                               [{file,\"src/rabbit_queue_index.erl\"},\r\n                                {line,1093}]},\r\n                              {rabbit_queue_index,parse_segment_entries,3,\r\n                               [{file,\"src/rabbit_queue_index.erl\"},\r\n                                {line,1077}]},\r\n                              {rabbit_queue_index,segment_entries_foldr,3,\r\n                               [{file,\"src/rabbit_queue_index.erl\"},\r\n                                {line,1044}]},\r\n                              {rabbit_queue_index,scan_segments,3,\r\n                               [{file,\"src/rabbit_queue_index.erl\"},\r\n                                {line,681}]},\r\n                              {rabbit_queue_index,queue_index_walker_reader,\r\n                               2,\r\n                               [{file,\"src/rabbit_queue_index.erl\"},\r\n                                {line,668}]},\r\n                              {rabbit_queue_index,\r\n                               '-queue_index_walker/1-fun-0-',2,\r\n                               [{file,\"src/rabbit_queue_index.erl\"},\r\n                                {line,649}]},\r\n                              {worker_pool_worker,handle_cast,2,\r\n                               [{file,\"src/worker_pool_worker.erl\"},\r\n                                {line,125}]},\r\n                              {gen_server2,handle_msg,2,\r\n                               [{file,\"src/gen_server2.erl\"},{line,1049}]}]},\r\n                            {gen_server2,call,[<0.253.0>,out,infinity]}},\r\n                           {child,undefined,msg_store_persistent,\r\n                            {rabbit_msg_store,start_link,\r\n                             [msg_store_persistent,\r\n                              \"/var/lib/rabbitmq/mnesia/rabbit@rabbit1\",[],\r\n                              {#Fun<rabbit_queue_index.2.122150458>,\r\n                               {start,\r\n                                [{resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.vendor_normalization_exception_queue_2\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"Reindexing.Partitions\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.alpha_key_master_vendor_aggregation_exception_queue\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.vendor_normalization_exception_queue_1\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.alpha_key_master_vendor_aggregation_queue\">>},\r\n                                 {resource,<<\"/\">>,queue,<<\"celery\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.vendorqueue\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"Reindex_old.Partitions\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"federal.batch.balancer\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"federal.batch.processor\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"federal.archive.forwarder\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"importprocessor.bulk\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"Reindex_old.Partitions.Errors\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.vendor_normalization_exception_queue\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.alphakeyaggregationexceptionqueue\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.alpha_key_master_vendor_aggregation_queue_temp\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"Reindexing.Command\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.exceptionqueue\">>},\r\n                                 {resource,<<\"/\">>,queue,<<\"brainiac.all\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.alpha_key_master_vendor_aggregation_exception_queue_old\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.alphakeyqueue\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.vendor_normalization_queue\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"Reindexing.Partitions.Errors\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"Reindex_old.Command\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.normalized_vendor_aggregation_exception_queue\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.normalized_vendor_aggregation_queue\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.normalized_vendor_aggregation_queue_temp\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"importprocessor.filedownload\">>},\r\n                                 {resource,<<\"/\">>,queue,\r\n                                  <<\"mvi.alphakeyaggregationqueue\">>}]}}]},\r\n                            transient,4294967295,worker,\r\n                            [rabbit_msg_store]}}}},\r\n                        [{rabbit_variable_queue,start_msg_store,2,\r\n                          [{file,\"src/rabbit_variable_queue.erl\"},{line,458}]},\r\n                         {rabbit_variable_queue,start,1,\r\n                          [{file,\"src/rabbit_variable_queue.erl\"},{line,440}]},\r\n                         {rabbit_priority_queue,start,1,\r\n                          [{file,\"src/rabbit_priority_queue.erl\"},{line,92}]},\r\n                         {rabbit_amqqueue,recover,0,\r\n                          [{file,\"src/rabbit_amqqueue.erl\"},{line,236}]},\r\n                         {rabbit,recover,0,\r\n                          [{file,\"src/rabbit.erl\"},{line,546}]},\r\n                         {rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,\r\n                          [{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},\r\n                         {rabbit_boot_steps,run_step,2,\r\n                          [{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},\r\n                         {rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,\r\n                          [{file,\"src/rabbit_boot_steps.erl\"},{line,26}]}]}}}}\r\n      in function  application_master:init/4 (application_master.erl, line 134)\r\n    ancestors: [<0.167.0>]\r\n    messages: [{'EXIT',<0.169.0>,normal}]\r\n    links: [<0.167.0>,<0.7.0>]\r\n    dictionary: []\r\n    trap_exit: true\r\n    status: running\r\n    heap_size: 4185\r\n    stack_size: 27\r\n    reductions: 352\r\n  neighbours:\r\n\r\n=SUPERVISOR REPORT==== 11-May-2016::16:47:34 ===\r\n     Supervisor: {local,worker_pool_sup}\r\n     Context:    child_terminated\r\n     Reason:     {{case_clause,undefined},\r\n                  [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                   {rabbit_queue_index,parse_segment_entries,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                   {rabbit_queue_index,segment_entries_foldr,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                   {rabbit_queue_index,scan_segments,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                   {rabbit_queue_index,queue_index_walker_reader,2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                   {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                   {worker_pool_worker,handle_cast,2,\r\n                       [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                   {gen_server2,handle_msg,2,\r\n                       [{file,\"src/gen_server2.erl\"},{line,1049}]}]}\r\n     Offender:   [{pid,<0.193.0>},\r\n                  {id,6},\r\n                  {mfargs,{worker_pool_worker,start_link,[worker_pool]}},\r\n                  {restart_type,transient},\r\n                  {shutdown,4294967295},\r\n                  {child_type,worker}]\r\n\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: rabbit\r\n    exited: {bad_return,\r\n             {{rabbit,start,[normal,[]]},\r\n              {'EXIT',\r\n               {{badmatch,\r\n                 {error,\r\n                  {{{{case_clause,undefined},\r\n                     [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                      {rabbit_queue_index,parse_segment_entries,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                      {rabbit_queue_index,segment_entries_foldr,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                      {rabbit_queue_index,scan_segments,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                      {rabbit_queue_index,queue_index_walker_reader,2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                      {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                      {worker_pool_worker,handle_cast,2,\r\n                       [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                      {gen_server2,handle_msg,2,\r\n                       [{file,\"src/gen_server2.erl\"},{line,1049}]}]},\r\n                    {gen_server2,call,[<0.253.0>,out,infinity]}},\r\n                   {child,undefined,msg_store_persistent,\r\n                    {rabbit_msg_store,start_link,\r\n                     [msg_store_persistent,\r\n                      \"/var/lib/rabbitmq/mnesia/rabbit@rabbit1\",[],\r\n                      {#Fun<rabbit_queue_index.2.122150458>,\r\n                       {start,\r\n                        [{resource,<<\"/\">>,queue,\r\n                          <<\"mvi.vendor_normalization_exception_queue_2\">>},\r\n                         {resource,<<\"/\">>,queue,<<\"Reindexing.Partitions\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.alpha_key_master_vendor_aggregation_exception_queue\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.vendor_normalization_exception_queue_1\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.alpha_key_master_vendor_aggregation_queue\">>},\r\n                         {resource,<<\"/\">>,queue,<<\"celery\">>},\r\n                         {resource,<<\"/\">>,queue,<<\"mvi.vendorqueue\">>},\r\n                         {resource,<<\"/\">>,queue,<<\"Reindex_old.Partitions\">>},\r\n                         {resource,<<\"/\">>,queue,<<\"federal.batch.balancer\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"federal.batch.processor\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"federal.archive.forwarder\">>},\r\n                         {resource,<<\"/\">>,queue,<<\"importprocessor.bulk\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"Reindex_old.Partitions.Errors\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.vendor_normalization_exception_queue\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.alphakeyaggregationexceptionqueue\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.alpha_key_master_vendor_aggregation_queue_temp\">>},\r\n                         {resource,<<\"/\">>,queue,<<\"Reindexing.Command\">>},\r\n                         {resource,<<\"/\">>,queue,<<\"mvi.exceptionqueue\">>},\r\n                         {resource,<<\"/\">>,queue,<<\"brainiac.all\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.alpha_key_master_vendor_aggregation_exception_queue_old\">>},\r\n                         {resource,<<\"/\">>,queue,<<\"mvi.alphakeyqueue\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.vendor_normalization_queue\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"Reindexing.Partitions.Errors\">>},\r\n                         {resource,<<\"/\">>,queue,<<\"Reindex_old.Command\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.normalized_vendor_aggregation_exception_queue\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.normalized_vendor_aggregation_queue\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.normalized_vendor_aggregation_queue_temp\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"importprocessor.filedownload\">>},\r\n                         {resource,<<\"/\">>,queue,\r\n                          <<\"mvi.alphakeyaggregationqueue\">>}]}}]},\r\n                    transient,4294967295,worker,\r\n                    [rabbit_msg_store]}}}},\r\n                [{rabbit_variable_queue,start_msg_store,2,\r\n                  [{file,\"src/rabbit_variable_queue.erl\"},{line,458}]},\r\n                 {rabbit_variable_queue,start,1,\r\n                  [{file,\"src/rabbit_variable_queue.erl\"},{line,440}]},\r\n                 {rabbit_priority_queue,start,1,\r\n                  [{file,\"src/rabbit_priority_queue.erl\"},{line,92}]},\r\n                 {rabbit_amqqueue,recover,0,\r\n                  [{file,\"src/rabbit_amqqueue.erl\"},{line,236}]},\r\n                 {rabbit,recover,0,[{file,\"src/rabbit.erl\"},{line,546}]},\r\n                 {rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,\r\n                  [{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},\r\n                 {rabbit_boot_steps,run_step,2,\r\n                  [{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},\r\n                 {rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,\r\n                  [{file,\"src/rabbit_boot_steps.erl\"},{line,26}]}]}}}}\r\n    type: temporary\r\n\r\n=SUPERVISOR REPORT==== 11-May-2016::16:47:34 ===\r\n     Supervisor: {local,worker_pool_sup}\r\n     Context:    child_terminated\r\n     Reason:     {{case_clause,undefined},\r\n                  [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                   {rabbit_queue_index,parse_segment_entries,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                   {rabbit_queue_index,segment_entries_foldr,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                   {rabbit_queue_index,scan_segments,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                   {rabbit_queue_index,queue_index_walker_reader,2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                   {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                   {worker_pool_worker,handle_cast,2,\r\n                       [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                   {gen_server2,handle_msg,2,\r\n                       [{file,\"src/gen_server2.erl\"},{line,1049}]}]}\r\n     Offender:   [{pid,<0.202.0>},\r\n                  {id,15},\r\n                  {mfargs,{worker_pool_worker,start_link,[worker_pool]}},\r\n                  {restart_type,transient},\r\n                  {shutdown,4294967295},\r\n                  {child_type,worker}]\r\n\r\n\r\n=SUPERVISOR REPORT==== 11-May-2016::16:47:34 ===\r\n     Supervisor: {local,worker_pool_sup}\r\n     Context:    child_terminated\r\n     Reason:     {{case_clause,undefined},\r\n                  [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                   {rabbit_queue_index,parse_segment_entries,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                   {rabbit_queue_index,segment_entries_foldr,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                   {rabbit_queue_index,scan_segments,3,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                   {rabbit_queue_index,queue_index_walker_reader,2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                   {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                       [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                   {worker_pool_worker,handle_cast,2,\r\n                       [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                   {gen_server2,handle_msg,2,\r\n                       [{file,\"src/gen_server2.erl\"},{line,1049}]}]}\r\n     Offender:   [{pid,<0.200.0>},\r\n                  {id,13},\r\n                  {mfargs,{worker_pool_worker,start_link,[worker_pool]}},\r\n                  {restart_type,transient},\r\n                  {shutdown,4294967295},\r\n                  {child_type,worker}]\r\n\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: ranch\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: rabbitmq_web_dispatch\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: webmachine\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: mochiweb\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: amqp_client\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: rabbit_common\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: os_mon\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: xmerl\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: ssl\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: compiler\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: mnesia\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: public_key\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: crypto\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: asn1\r\n    exited: stopped\r\n    type: temporary\r\n\r\n\r\nBOOT FAILED\r\n===========\r\n\r\nError description:\r\n   {could_not_start,rabbit,\r\n       {{badmatch,\r\n            {error,\r\n                {{{{case_clause,undefined},\r\n                   [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                    {rabbit_queue_index,parse_segment_entries,3,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                    {rabbit_queue_index,segment_entries_foldr,3,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                    {rabbit_queue_index,scan_segments,3,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                    {rabbit_queue_index,queue_index_walker_reader,2,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                    {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                    {worker_pool_worker,handle_cast,2,\r\n                        [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                    {gen_server2,handle_msg,2,\r\n                        [{file,\"src/gen_server2.erl\"},{line,1049}]}]},\r\n                  {gen_server2,call,[<0.253.0>,out,infinity]}},\r\n                 {child,undefined,msg_store_persistent,\r\n                     {rabbit_msg_store,start_link,\r\n                         [msg_store_persistent,\r\n                          \"/var/lib/rabbitmq/mnesia/rabbit@rabbit1\",[],\r\n                          {#Fun<rabbit_queue_index.2.122150458>,\r\n                           {start,\r\n                               [{resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.vendor_normalization_exception_queue_2\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindexing.Partitions\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alpha_key_master_vendor_aggregation_exception_queue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.vendor_normalization_exception_queue_1\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alpha_key_master_vendor_aggregation_queue\">>},\r\n                                {resource,<<\"/\">>,queue,<<\"celery\">>},\r\n                                {resource,<<\"/\">>,queue,<<\"mvi.vendorqueue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindex_old.Partitions\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"federal.batch.balancer\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"federal.batch.processor\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"federal.archive.forwarder\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"importprocessor.bulk\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindex_old.Partitions.Errors\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.vendor_normalization_exception_queue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alphakeyaggregationexceptionqueue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alpha_key_master_vendor_aggregation_queue_temp\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindexing.Command\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.exceptionqueue\">>},\r\n                                {resource,<<\"/\">>,queue,<<\"brainiac.all\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alpha_key_master_vendor_aggregation_exception_queue_old\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alphakeyqueue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.vendor_normalization_queue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindexing.Partitions.Errors\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindex_old.Command\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.normalized_vendor_aggregation_exception_queue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.normalized_vendor_aggregation_queue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.normalized_vendor_aggregation_queue_temp\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"importprocessor.filedownload\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alphakeyaggregationqueue\">>}]}}]},\r\n                     transient,4294967295,worker,\r\n                     [rabbit_msg_store]}}}},\r\n        [{rabbit_variable_queue,start_msg_store,2,\r\n             [{file,\"src/rabbit_variable_queue.erl\"},{line,458}]},\r\n         {rabbit_variable_queue,start,1,\r\n             [{file,\"src/rabbit_variable_queue.erl\"},{line,440}]},\r\n         {rabbit_priority_queue,start,1,\r\n             [{file,\"src/rabbit_priority_queue.erl\"},{line,92}]},\r\n         {rabbit_amqqueue,recover,0,\r\n             [{file,\"src/rabbit_amqqueue.erl\"},{line,236}]},\r\n         {rabbit,recover,0,[{file,\"src/rabbit.erl\"},{line,546}]},\r\n         {rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,\r\n             [{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},\r\n         {rabbit_boot_steps,run_step,2,\r\n             [{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},\r\n         {rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,\r\n             [{file,\"src/rabbit_boot_steps.erl\"},{line,26}]}]}}\r\n\r\nLog files (may contain more information):\r\n   tty\r\n   tty\r\n\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: inets\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\n    application: syntax_tools\r\n    exited: stopped\r\n    type: temporary\r\n\r\n=INFO REPORT==== 11-May-2016::16:47:34 ===\r\nError description:\r\n   {could_not_start,rabbit,\r\n       {{badmatch,\r\n            {error,\r\n                {{{{case_clause,undefined},\r\n                   [{rabbit_queue_index,add_segment_relseq_entry,3,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},\r\n                    {rabbit_queue_index,parse_segment_entries,3,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},\r\n                    {rabbit_queue_index,segment_entries_foldr,3,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},\r\n                    {rabbit_queue_index,scan_segments,3,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,681}]},\r\n                    {rabbit_queue_index,queue_index_walker_reader,2,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,668}]},\r\n                    {rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,\r\n                        [{file,\"src/rabbit_queue_index.erl\"},{line,649}]},\r\n                    {worker_pool_worker,handle_cast,2,\r\n                        [{file,\"src/worker_pool_worker.erl\"},{line,125}]},\r\n                    {gen_server2,handle_msg,2,\r\n                        [{file,\"src/gen_server2.erl\"},{line,1049}]}]},\r\n                  {gen_server2,call,[<0.253.0>,out,infinity]}},\r\n                 {child,undefined,msg_store_persistent,\r\n                     {rabbit_msg_store,start_link,\r\n                         [msg_store_persistent,\r\n                          \"/var/lib/rabbitmq/mnesia/rabbit@rabbit1\",[],\r\n                          {#Fun<rabbit_queue_index.2.122150458>,\r\n                           {start,\r\n                               [{resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.vendor_normalization_exception_queue_2\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindexing.Partitions\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alpha_key_master_vendor_aggregation_exception_queue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.vendor_normalization_exception_queue_1\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alpha_key_master_vendor_aggregation_queue\">>},\r\n                                {resource,<<\"/\">>,queue,<<\"celery\">>},\r\n                                {resource,<<\"/\">>,queue,<<\"mvi.vendorqueue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindex_old.Partitions\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"federal.batch.balancer\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"federal.batch.processor\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"federal.archive.forwarder\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"importprocessor.bulk\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindex_old.Partitions.Errors\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.vendor_normalization_exception_queue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alphakeyaggregationexceptionqueue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alpha_key_master_vendor_aggregation_queue_temp\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindexing.Command\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.exceptionqueue\">>},\r\n                                {resource,<<\"/\">>,queue,<<\"brainiac.all\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alpha_key_master_vendor_aggregation_exception_queue_old\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alphakeyqueue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.vendor_normalization_queue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindexing.Partitions.Errors\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"Reindex_old.Command\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.normalized_vendor_aggregation_exception_queue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.normalized_vendor_aggregation_queue\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.normalized_vendor_aggregation_queue_temp\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"importprocessor.filedownload\">>},\r\n                                {resource,<<\"/\">>,queue,\r\n                                    <<\"mvi.alphakeyaggregationqueue\">>}]}}]},\r\n                     transient,4294967295,worker,\r\n                     [rabbit_msg_store]}}}},\r\n        [{rabbit_variable_queue,start_msg_store,2,\r\n             [{file,\"src/rabbit_variable_queue.erl\"},{line,458}]},\r\n         {rabbit_variable_queue,start,1,\r\n             [{file,\"src/rabbit_variable_queue.erl\"},{line,440}]},\r\n         {rabbit_priority_queue,start,1,\r\n             [{file,\"src/rabbit_priority_queue.erl\"},{line,92}]},\r\n         {rabbit_amqqueue,recover,0,\r\n             [{file,\"src/rabbit_amqqueue.erl\"},{line,236}]},\r\n         {rabbit,recover,0,[{file,\"src/rabbit.erl\"},{line,546}]},\r\n         {rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,\r\n             [{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},\r\n         {rabbit_boot_steps,run_step,2,\r\n             [{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},\r\n         {rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,\r\n             [{file,\"src/rabbit_boot_steps.erl\"},{line,26}]}]}}\r\n\r\nLog files (may contain more information):\r\n   tty\r\n   tty\r\n\r\n{\"init terminating in do_boot\",{could_not_start,rabbit,{{badmatch,{error,{{{{case_clause,undefined},[{rabbit_queue_index,add_segment_relseq_entry,3,[{file,\"src/rabbit_queue_index.erl\"},{line,1093}]},{rabbit_queue_index,parse_segment_entries,3,[{file,\"src/rabbit_queue_index.erl\"},{line,1077}]},{rabbit_queue_index,segment_entries_foldr,3,[{file,\"src/rabbit_queue_index.erl\"},{line,1044}]},{rabbit_queue_index,scan_segments,3,[{file,\"src/rabbit_queue_index.erl\"},{line,681}]},{rabbit_queue_index,queue_index_walker_reader,2,[{file,\"src/rabbit_queue_index.erl\"},{line,668}]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,\"src/rabbit_queue_index.erl\"},{line,649}]},{worker_pool_worker,handle_cast,2,[{file,\"src/worker_pool_worker.erl\"},{line,125}]},{gen_server2,handle_msg,2,[{file,\"src/gen_server2.erl\"},{line,1049}]}]},{gen_server2,call,[<0.253.0>,out,infinity]}},{child,undefined,msg_store_persistent,{rabbit_msg_store,start_link,[msg_store_persistent,\"/var/lib/rabbitmq/mnesia/rabbit@rabbit1\",[],{#Fun<rabbit_queue_index.2.122150458>,{start,[{resource,<<\"/\">>,queue,<<\"mvi.vendor_normalization_exception_queue_2\">>},{resource,<<\"/\">>,queue,<<\"Reindexing.Partitions\">>},{resource,<<\"/\">>,queue,<<\"mvi.alpha_key_master_vendor_aggregation_exception_queue\">>},{resource,<<\"/\">>,queue,<<\"mvi.vendor_normalization_exception_queue_1\">>},{resource,<<\"/\">>,queue,<<\"mvi.alpha_key_master_vendor_aggregation_queue\">>},{resource,<<\"/\">>,queue,<<\"celery\">>},{resource,<<\"/\">>,queue,<<\"mvi.vendorqueue\">>},{resource,<<\"/\">>,queue,<<\"Reindex_old.Partitions\">>},{resource,<<\"/\">>,queue,<<\"federal.batch.balancer\">>},{resource,<<\"/\">>,queue,<<\"federal.batch.processor\">>},{resource,<<\"/\">>,queue,<<\"federal.archive.forwarder\">>},{resource,<<\"/\">>,queue,<<\"importprocessor.bulk\">>},{resource,<<\"/\">>,queue,<<\"Reindex_old.Partitions.Errors\">>},{resource,<<\"/\">>,queue,<<\"mvi.vendor_normalization_exception_queue\">>},{resource,<<\"/\">>,queue,<<\"mvi.alphakeyaggregationexceptionqueue\">>},{resource,<<\"/\">>,queue,<<\"mvi.alpha_key_master_vendor_aggregation_queue_temp\">>},{resource,<<\"/\">>,queue,<<\"Reindexing.Command\">>},{resource,<<\"/\">>,queue,<<\"mvi.exceptionqueue\">>},{resource,<<\"/\">>,queue,<<\"brainiac.all\">>},{resource,<<\"/\">>,queue,<<\"mvi.alpha_key_master_vendor_aggregation_exception_queue_old\">>},{resource,<<\"/\">>,queue,<<\"mvi.alphakeyqueue\">>},{resource,<<\"/\">>,queue,<<\"mvi.vendor_normalization_queue\">>},{resource,<<\"/\">>,queue,<<\"Reindexing.Partitions.Errors\">>},{resource,<<\"/\">>,queue,<<\"Reindex_old.Command\">>},{resource,<<\"/\">>,queue,<<\"mvi.normalized_vendor_aggregation_exception_queue\">>},{resource,<<\"/\">>,queue,<<\"mvi.normalized_vendor_aggregation_queue\">>},{resource,<<\"/\">>,queue,<<\"mvi.normalized_vendor_aggregation_queue_temp\">>},{resource,<<\"/\">>,queue,<<\"importprocessor.filedownload\">>},{resource,<<\"/\">>,queue,<<\"mvi.alphakeyaggregationqueue\">>}]}}]},transient,4294967295,worker,[rabbit_msg_store]}}}},[{rabbit_variable_queue,start_msg_store,2,[{file,\"src/rabbit_variable_queue.erl\"},{line,458}]},{rabbit_variable_queue,start,1,[{file,\"src/rabbit_variable_queue.erl\"},{line,440}]},{rabbit_priority_queue,start,1,[{file,\"src/rabbit_priority_queue.erl\"},{line,92}]},{rabbit_amqqueue,recover,0,[{file,\"src/rabbit_amqqueue.erl\"},{line,236}]},{rabbit,recover,0,[{file,\"src/rabbit.erl\"},{line,546}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]}]}}}"}}