title: rabbitmq connections blocking but memory is below watermark
body: <p>we are use rabbitmq in our application,  two hours ago, one of our app server is blocked when try to connect to rabbitmq, after check rabbitmq server ,  we found one node's memory is over watermark, a few minutes later, this node is down. after restart this node, the whole cluster sames work fine, but i notice there's a lot of connection in blocking and blocked state from web management，but use  <code>rabbitmqctl list_connections pid name peer_address state</code> in all nodes shows there is no connection in blocking/blocked…so this really make me confuse: </p>
  
  <ol>
  <li>after  one node of whole cluster over watermark, but other node is
  work fine, my application can't connect to rabbitmq cluster?  ps: 
  we use spring.amqp &amp; spring-rabbit with version 1.1.0.RELEASE</li>
  <li>node will down for what reason when over watermark?     </li>
  <li>why after restart node, there is still blocking connection, but with rabbitmqctl they
  all in running state?</li>
  </ol>
  
  <p>here is some logs from my rabbitmq server:</p>
  
  <pre><code>=INFO REPORT==== 1-Mar-2013::19:36:21 ===
  vm_memory_high_watermark clear. Memory used:1656590680 allowed:1658778419
  
  =INFO REPORT==== 1-Mar-2013::19:36:21 ===
  alarm_handler: {clear,{resource_limit,memory,rabbit@cos22}}
  </code></pre>
  
  <p>when i try to close blocked connection from web management, it goes error:</p>
  
  <pre><code>=INFO REPORT==== 1-Mar-2013::20:55:24 ===
  Closing connection &lt;0.17197.115&gt; because "Closed via management plugin"
  
  =ERROR REPORT==== 1-Mar-2013::20:55:24 ===
  webmachine error: path="/api/connections/10.64.13.200%3A45891%20-%3E%2010.64.12.226%3A5672"
  {throw,
  {error,{not_a_connection_pid,&lt;0.17197.115&gt;}},
  [{rabbit_networking,close_connection,2,
       [{file,"src/rabbit_networking.erl"},{line,317}]},
   {rabbit_mgmt_wm_connection,delete_resource,2,
       [{file,"rabbitmq-management/src/rabbit_mgmt_wm_connection.erl"},
        {line,52}]},
   {webmachine_resource,resource_call,3,
       [{file,
            "webmachine-wrapper/webmachine-git/src/webmachine_resource.erl"},
        {line,169}]},
   {webmachine_resource,do,3,
       [{file,
            "webmachine-wrapper/webmachine-git/src/webmachine_resource.erl"},
        {line,128}]},
   {webmachine_decision_core,resource_call,1,
       [{file,
            "webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl"},
        {line,48}]},
   {webmachine_decision_core,decision,1,
       [{file,
            "webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl"},
        {line,416}]},
   {webmachine_decision_core,handle_request,2,
       [{file,
            "webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl"},
        {line,33}]},
   {rabbit_webmachine,'-makeloop/1-fun-0-',3,
       [{file,"rabbitmq-mochiweb/src/rabbit_webmachine.erl"},{line,75}]}]}
  </code></pre>
  
  <p>use rabbitmqctl shows all in running state:</p>
  
  <pre><code>rabbitmqctl list_connections pid name peer_address state
  Listing connections ...
  &lt;rabbit@cos23.1.1271.51&gt;        10.64.13.197:57321 -&gt; 10.64.12.225:5672 10.64.13.197    running
  &lt;rabbit@cos23.1.1100.51&gt;        10.64.13.196:57240 -&gt; 10.64.12.225:5672 10.64.13.196    running
  &lt;rabbit@cos23.1.1056.51&gt;        10.64.12.196:58608 -&gt; 10.64.12.225:5672 10.64.12.196    running
  &lt;rabbit@cos23.1.1079.51&gt;        10.64.11.235:48962 -&gt; 10.64.12.225:5672 10.64.11.235    running
  &lt;rabbit@cos23.1.1419.51&gt;        10.64.13.228:49857 -&gt; 10.64.12.225:5672 10.64.13.228    running
  &lt;rabbit@cos23.1.1049.51&gt;        10.64.11.193:36387 -&gt; 10.64.12.225:5672 10.64.11.193    running
  &lt;rabbit@cos23.1.1159.51&gt;        10.64.10.123:52017 -&gt; 10.64.12.225:5672 10.64.10.123    running
  &lt;rabbit@cos23.1.26289.45&gt;       10.64.12.247:38504 -&gt; 10.64.12.225:5672 10.64.12.247    running
  &lt;rabbit@cos23.1.1121.51&gt;        10.64.10.29:51483 -&gt; 10.64.12.225:5672  10.64.10.29     running
  &lt;rabbit@cos23.1.1067.51&gt;        10.64.11.234:50244 -&gt; 10.64.12.225:5672 10.64.11.234    running
  &lt;rabbit@cos23.1.1149.51&gt;        10.64.11.178:33795 -&gt; 10.64.12.225:5672 10.64.11.178    running
  &lt;rabbit@cos23.1.1136.51&gt;        10.64.10.28:39557 -&gt; 10.64.12.225:5672  10.64.10.28     running
  &lt;rabbit@cos23.1.1370.51&gt;        10.64.13.233:38766 -&gt; 10.64.12.225:5672 10.64.13.233    running
  &lt;rabbit@cos23.1.1388.51&gt;        10.64.13.229:50932 -&gt; 10.64.12.225:5672 10.64.13.229    running
  &lt;rabbit@cos23.1.1254.51&gt;        10.64.13.241:49311 -&gt; 10.64.12.225:5672 10.64.13.241    running
  &lt;rabbit@cos23.1.1031.51&gt;        10.64.11.195:39455 -&gt; 10.64.12.225:5672 10.64.11.195    running
  &lt;rabbit@cos23.1.1038.51&gt;        10.64.10.27:58938 -&gt; 10.64.12.225:5672  10.64.10.27     running
  &lt;rabbit@cos23.1.1167.51&gt;        10.64.13.240:37777 -&gt; 10.64.12.225:5672 10.64.13.240    running
  &lt;rabbit@cos23.1.1442.51&gt;        10.64.10.130:37251 -&gt; 10.64.12.225:5672 10.64.10.130    running
  &lt;rabbit@cos22.3.2659.0&gt; 10.64.13.200:54840 -&gt; 10.64.12.226:5672 10.64.13.200    running
  ...done.
  </code></pre>
  
  <p>and there is a connection with a lot of channel is in blocked state, but i can't find this connection by use rabbitctl list_connections:</p>
  
  <pre><code>AMQP 0-9-1  
  10.64.13.200:45891 -&gt; 10.64.12.226:5672
  rabbit@cos22    0B/s
  (49.2MB total)
  0B/s
  (2.4MB total)
  0s  60920
  </code></pre>
  
  <p>thanks a lot for any help and suggestion.</p>