title: AWS classic LB changing IPs/dropping connections results in lost messages on RabbitMQ
body: <p>I run a rabbit HA cluster with 3 nodes and a classic AWS load-balancer(LB) in front of them. There are two apps, one that publishes and the other one that consumes through the LB. 
  <a href="https://i.stack.imgur.com/vJsfo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vJsfo.png" alt="enter image description here"></a>
  When publisher app starts sending 3 million messages, after short period of time its connection is put into Flow Control state. After the publishing is finished, in publisher app logs I can see that all 3 million messages are sent. On the other hand in consumer app log I can only see 500K - 1M messages (varies between runs), which means that the large number of messages is lost.</p>
  
  <p>So what is happening is that in the middle of a run, classic LB decides to change its IP address or drop connections, thus loosing a lot of  messages (see my update for more details).</p>
  
  <p>The issue does not occur if I skip LB and hit the nodes directly, doing load-balancing on app side. Of course in this case I lose all the benefits of ELB.</p>
  
  <p>My question are:</p>
  
  <ul>
  <li>Why is LB changing IP addresses and dropping connections, is that related to high message rate from publisher or Flow Control state?</li>
  <li>How to configure LB, so that this issue doesn't occur?</li>
  </ul>
  
  <p><strong>UPDATE:</strong></p>
  
  <p>This is my understanding what is happening:
  I use AMQP 0-9-1 and publish without '<em>publish confirms</em>', so message is considered sent as soon as it's put on a wire. Also, the connection on rabbitmq node is between LB and a node, not Publisher app and a node. </p>
  
  <ol>
  <li><p>Before the communication enters Flow Control, messages are passed from LB to a node immediately
  <a href="https://i.stack.imgur.com/O35Ld.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/O35Ld.png" alt="enter image description here"></a></p></li>
  <li><p>Then the connection between LB and a node enters Flow Control, Publisher App connection is not blocked and thus it continues to publish at the same rate. That causes messages to pile up on LB.
  <a href="https://i.stack.imgur.com/KrBpM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KrBpM.png" alt="enter image description here"></a></p></li>
  <li><p>Then LB decides to change IP(s) or drop the connection for whatever reasons and create a new one, causing all the piled messages to be lost. This is clearly visible from the RabbitMQ logs:</p>
  
  <p>=WARNING REPORT==== 6-Jan-2018::10:35:50 ===
  closing AMQP connection &lt;0.30342.375> (10.1.1.250:29564 -> 10.1.1.223:5672):
  client unexpectedly closed TCP connection</p>
  
  <p>=INFO REPORT==== 6-Jan-2018::10:35:51 ===
  accepting AMQP connection &lt;0.29123.375> (10.1.1.22:1886 -> 10.1.1.223:5672)</p></li>
  </ol>
  
  <p><a href="https://i.stack.imgur.com/Yqptw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Yqptw.png" alt="enter image description here"></a></p>