title: Spring AMQP v1.4.2 - Rabbit reconnection issue on network failure
body: <p>I am testing the following scenario in Spring AMQP v1.4.2 and it fails to reconnect after a network disruption:</p>
  <ol>
  <li>Start the spring application which consumes messages asyncly using rabbit:listener-container and rabbit:connection-factory (detailed configuration follows).</li>
  <li>The log shows that application is successfully receiving messages.</li>
  <li>Make RabbitMQ invisible to the app by dropping inbound network traffic on rabbit server: <code>sudo iptables -A INPUT -p tcp --destination-port 5672 -j DROP</code></li>
  <li>Wait for at least 3 minutes (for network connections to timeout).</li>
  <li>Fix the connection with: <code>sudo iptables -D INPUT -p tcp --destination-port 5672 -j DROP</code></li>
  <li>Wait for some time (even tried more than an hour) and no reconnection happens.</li>
  <li>Restart the application and it starts receiving messages again, which means network was back to normal.</li>
  </ol>
  <p>I have also tested the same scenario with VM network adapter disconnects instead of iptables drop and the same thing happens, i.e. no automatic reconnection. Interestingly enough, when I try iptables <em>REJECT</em>, instead of DROP, it works as expected and the app restarts as soon as I remove the reject rule but I think reject is more like a server failure than a network failure.</p>
  <p>According to the <a href="http://docs.spring.io/spring-amqp/reference/html/amqp.html#async-listeners" rel="noreferrer">reference document</a>:</p>
  <blockquote>
  <p>If a MessageListener fails because of a business exception, the exception is handled by the message listener container and then it goes back to listening for another message. If the failure is caused by a dropped connection (not a business exception), then the consumer that is collecting messages for the listener has to be cancelled and restarted. <strong>The SimpleMessageListenerContainer handles this seamlessly, and it leaves a log to say that the listener is being restarted.</strong> In fact it loops endlessly trying to restart the consumer, and only if the consumer is very badly behaved indeed will it give up. One side effect is that if the broker is down when the container starts, it will just keep trying until a connection can be established.</p>
  </blockquote>
  <p>This is the log that I get about a minute after disconnection:</p>
  <pre><code>    2015-01-16 14:00:42,433 WARN  [SimpleAsyncTaskExecutor-5] org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer Consumer raised exception, processing can restart if the connection factory supports it
  com.rabbitmq.client.ShutdownSignalException: connection error
      at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:717) ~[amqp-client-3.4.2.jar:na]
      at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:707) ~[amqp-client-3.4.2.jar:na]
      at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:565) ~[amqp-client-3.4.2.jar:na]
      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]
  Caused by: java.io.EOFException: null
      at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290) ~[na:1.7.0_55]
      at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95) ~[amqp-client-3.4.2.jar:na]
      at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:139) ~[amqp-client-3.4.2.jar:na]
      at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:534) ~[amqp-client-3.4.2.jar:na]
      ... 1 common frames omitted
  </code></pre>
  <p>And I get this log message a few seconds after the reconnection:</p>
  <pre><code>2015-01-16 14:18:14,551 WARN  [SimpleAsyncTaskExecutor-2] org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection timed out
  </code></pre>
  <p><strong>UPDATE:</strong> Quite strangely, when I enable DEBUG logging on org.springframework.amqp package, the reconnection happens successfully and I cannot reproduce the issue anymore!</p>
  <p>Without debug logging enabled, I tried to debug the spring AMQP code. I observed that soon after iptables drop is removed, <code>SimpleMessageListenerContainer.doStop()</code> method is called which it turn calls shutdown() and cancels all of the channels. I also got this log message when I put a breakpoint on doStop() which seems to be related to the cause:</p>
  <pre><code>2015-01-20 15:28:44,200 ERROR [pool-1-thread-16] org.springframework.amqp.rabbit.connection.CachingConnectionFactory Channel shutdown: channel error; protocol method: #method&lt;channel.close&gt;(reply-code=405, reply-text=RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'e4288669-2422-40e6-a2ee-b99542509273' in vhost '/', class-id=50, method-id=10)
  2015-01-20 15:28:44,243 WARN  [SimpleAsyncTaskExecutor-3] org.springframework.amqp.rabbit.listener.BlockingQueueConsumer Failed to declare queue:e4288669-2422-40e6-a2ee-b99542509273
  2015-01-20 15:28:44,243 WARN  [SimpleAsyncTaskExecutor-3] org.springframework.amqp.rabbit.listener.BlockingQueueConsumer Queue declaration failed; retries left=0
  org.springframework.amqp.rabbit.listener.BlockingQueueConsumer$DeclarationException: Failed to declare queue(s):[e4288669-2422-40e6-a2ee-b99542509273]
      at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:486) ~[spring-rabbit-1.4.2.RELEASE.jar:na]
      at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:401) ~[spring-rabbit-1.4.2.RELEASE.jar:na]
      at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1022) [spring-rabbit-1.4.2.RELEASE.jar:na]
      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]
  2015-01-20 15:28:49,245 ERROR [pool-1-thread-16] org.springframework.amqp.rabbit.connection.CachingConnectionFactory Channel shutdown: channel error; protocol method: #method&lt;channel.close&gt;(reply-code=405, reply-text=RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'e4288669-2422-40e6-a2ee-b99542509273' in vhost '/', class-id=50, method-id=10)
  2015-01-20 15:28:49,283 WARN  [SimpleAsyncTaskExecutor-3] org.springframework.amqp.rabbit.listener.BlockingQueueConsumer Failed to declare queue:e4288669-2422-40e6-a2ee-b99542509273
  2015-01-20 15:28:49,300 ERROR [SimpleAsyncTaskExecutor-3] org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer Consumer received fatal exception on startup
  org.springframework.amqp.rabbit.listener.QueuesNotAvailableException: Cannot prepare queue for listener. Either the queue doesn't exist or the broker will not allow us to use it.
      at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:429) ~[spring-rabbit-1.4.2.RELEASE.jar:na]
      at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1022) ~[spring-rabbit-1.4.2.RELEASE.jar:na]
      at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]
  Caused by: org.springframework.amqp.rabbit.listener.BlockingQueueConsumer$DeclarationException: Failed to declare queue(s):[e4288669-2422-40e6-a2ee-b99542509273]
      at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:486) ~[spring-rabbit-1.4.2.RELEASE.jar:na]
      at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:401) ~[spring-rabbit-1.4.2.RELEASE.jar:na]
      ... 2 common frames omitted
  2015-01-20 15:28:49,301 ERROR [SimpleAsyncTaskExecutor-3] org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer Stopping container from aborted consumer
  </code></pre>
  <p><strong>UPDATE 2:</strong> After setting <code>requested-heartbeat</code> to 30 seconds, as suggested in an answer, the reconnection worked most of the times and succeeded in redefining the exclusive temporary queue, bound to a fanout exchange, but it still fails to reconnect occasionally.</p>
  <p>In the rare cases that it failed, I monitored RabbitMQ management console during the test and observed that a new connection was established (after the old connection was removed by timeout) but the exclusive temporary queue was not redefined after reconnection. Also the client was not receiving any messages. It is now really hard to reproduce the issue reliably as it happens less often. I have provided the full configuration below, now containing the queue declarations.</p>
  <p><strong>UPDATE 3:</strong> Even after replacing the exclusive temporary queue with an auto-delete named queue, the same behaviour occurs occasionally; i.e. the auto-delete named queue is not redefined after reconnection and no messages are being received until application is restarted.</p>
  <p>I would really appreciate a lot if someone can help me on this.</p>
  <p>Here is the spring AMQP configuration that I am relying on:</p>
  <pre><code>&lt;!-- Create a temporary exclusive queue to subscribe to the control exchange --&gt;
  &lt;rabbit:queue id=&quot;control-queue&quot;/&gt;
  
  &lt;!-- Bind the temporary queue to the control exchange --&gt;
  &lt;rabbit:fanout-exchange name=&quot;control&quot;&gt;
      &lt;rabbit:bindings&gt;
          &lt;rabbit:binding queue=&quot;control-queue&quot;/&gt;
      &lt;/rabbit:bindings&gt;
  &lt;/rabbit:fanout-exchange&gt;
  
  &lt;!-- Subscribe to the temporary queue --&gt;
  &lt;rabbit:listener-container connection-factory=&quot;connection-factory&quot;
                             acknowledge=&quot;none&quot;
                             concurrency=&quot;1&quot;
                             prefetch=&quot;1&quot;&gt;
      &lt;rabbit:listener queues=&quot;control-queue&quot; ref=&quot;controlQueueConsumer&quot;/&gt;
  
  &lt;/rabbit:listener-container&gt;
  
  &lt;rabbit:connection-factory id=&quot;connection-factory&quot;
                             username=&quot;${rabbit.username}&quot;
                             password=&quot;${rabbit.password}&quot;
                             host=&quot;${rabbit.host}&quot;
                             virtual-host=&quot;${rabbit.virtualhost}&quot;
                             publisher-confirms=&quot;true&quot; 
                             channel-cache-size=&quot;100&quot;
                             requested-heartbeat=&quot;30&quot; /&gt;
  
  &lt;rabbit:admin id=&quot;admin&quot; connection-factory=&quot;connection-factory&quot;/&gt;
  
  &lt;rabbit:queue id=&quot;qu0-id&quot; name=&quot;qu0&quot;&gt;
      &lt;rabbit:queue-arguments&gt;
          &lt;entry key=&quot;x-dead-letter-exchange&quot; value=&quot;dead-letter&quot;/&gt;
      &lt;/rabbit:queue-arguments&gt;
  &lt;/rabbit:queue&gt;
  
  &lt;rabbit:topic-exchange id=&quot;default-exchange&quot; name=&quot;default-ex&quot; declared-by=&quot;admin&quot;&gt;
      &lt;rabbit:bindings&gt;
          &lt;rabbit:binding queue=&quot;qu0&quot; pattern=&quot;p.0&quot;/&gt;
      &lt;/rabbit:bindings&gt;
  &lt;/rabbit:topic-exchange&gt;
  
  &lt;rabbit:listener-container connection-factory=&quot;connection-factory&quot;
                             acknowledge=&quot;manual&quot;
                             concurrency=&quot;4&quot;
                             prefetch=&quot;30&quot;&gt;
      &lt;rabbit:listener queues=&quot;qu0&quot; ref=&quot;queueConsumerComponent&quot;/&gt;
  &lt;/rabbit:listener-container&gt;
  </code></pre>