<p>I need some help with the correct setting for using the client connection to a ActiveMQ Artemis broker cluster.</p>
<p>My goal here is to run a performance test against what I believe is our Artemis cluster. While testing I would like to kill some nodes and expect the performance test to continue.</p>
<p>First the cluster setup:</p>
<p>We are about to migrate to the new Artemis Cloud and have deployed <a href="https://github.com/artemiscloud/activemq-artemis-operator" rel="nofollow noreferrer">the Artemis operator</a> in our Kubernetes cluster. It is configured to run with three nodes at the moment.</p>
<p><code>broker.xml</code>:</p>
<pre class="lang-xml prettyprint-override"><code>&lt;connectors&gt;
  &lt;connector name=&quot;artemis&quot;&gt;tcp://msc-test-ss-0.msc-test-hdls-svc.it.svc.cluster.local:61616&lt;/connector&gt;
&lt;/connectors&gt;

&lt;acceptors&gt;
  &lt;acceptor name=&quot;msc-test&quot;&gt;
    tcp://msc-test-ss-0.msc-test-hdls-svc.it.svc.cluster.local:61616?protocols=AMQP,CORE,HORNETQ,MQTT,OPENWIRE,STOMP;tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;useEpoll=true;amqpCredits=1000;amqpMinCredits=300
  &lt;/acceptor&gt;
&lt;/acceptors&gt;

&lt;cluster-user&gt;SOME_USER&lt;/cluster-user&gt;

&lt;cluster-password&gt;SOME_PASSWORD&lt;/cluster-password&gt;

&lt;broadcast-groups&gt;
  &lt;broadcast-group name=&quot;my-broadcast-group&quot;&gt;
    &lt;jgroups-file&gt;jgroups-ping.xml&lt;/jgroups-file&gt;
    &lt;jgroups-channel&gt;activemq_broadcast_channel&lt;/jgroups-channel&gt;
    &lt;connector-ref&gt;artemis&lt;/connector-ref&gt;
  &lt;/broadcast-group&gt;
&lt;/broadcast-groups&gt;

&lt;discovery-groups&gt;
  &lt;discovery-group name=&quot;my-discovery-group&quot;&gt;
    &lt;jgroups-file&gt;jgroups-ping.xml&lt;/jgroups-file&gt;
    &lt;jgroups-channel&gt;activemq_broadcast_channel&lt;/jgroups-channel&gt;
    &lt;refresh-timeout&gt;10000&lt;/refresh-timeout&gt;
  &lt;/discovery-group&gt;
&lt;/discovery-groups&gt;

&lt;cluster-connections&gt;
  &lt;cluster-connection name=&quot;my-cluster&quot;&gt;
    &lt;connector-ref&gt;artemis&lt;/connector-ref&gt;
    &lt;retry-interval&gt;1000&lt;/retry-interval&gt;
    &lt;retry-interval-multiplier&gt;2&lt;/retry-interval-multiplier&gt;
    &lt;max-retry-interval&gt;32000&lt;/max-retry-interval&gt;
    &lt;initial-connect-attempts&gt;20&lt;/initial-connect-attempts&gt;
    &lt;reconnect-attempts&gt;10&lt;/reconnect-attempts&gt;
    &lt;use-duplicate-detection&gt;true&lt;/use-duplicate-detection&gt;
    &lt;message-load-balancing&gt;ON_DEMAND&lt;/message-load-balancing&gt;
    &lt;max-hops&gt;1&lt;/max-hops&gt;
    &lt;discovery-group-ref discovery-group-name=&quot;my-discovery-group&quot;/&gt;
  &lt;/cluster-connection&gt;
&lt;/cluster-connections&gt;
</code></pre>
<p><code>jgroups-ping.xml</code>:</p>
<pre class="lang-xml prettyprint-override"><code>&lt;config xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;
        xmlns=&quot;urn:org:jgroups&quot;
        xsi:schemaLocation=&quot;urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd&quot;
        &gt;
    &lt;TCP bind_addr=&quot;${jgroups.bind_addr:site_local}&quot;
         bind_port=&quot;${jgroups.bind_port:7800}&quot;
         external_addr=&quot;${jgroups.external_addr}&quot;
         external_port=&quot;${jgroups.external_port}&quot;
         thread_pool.min_threads=&quot;0&quot;
         thread_pool.max_threads=&quot;200&quot;
         thread_pool.keep_alive_time=&quot;30000&quot;/&gt;
    &lt;RED/&gt;

    &lt;dns.DNS_PING
            dns_query=&quot;msc-test-ping-svc&quot;
            dns_record_type=&quot;${DNS_RECORD_TYPE:A}&quot; /&gt;

    &lt;MERGE3  min_interval=&quot;10000&quot;
             max_interval=&quot;30000&quot;/&gt;
    &lt;FD_SOCK2/&gt;
    &lt;FD_ALL3 timeout=&quot;40000&quot; interval=&quot;5000&quot; /&gt;
    &lt;VERIFY_SUSPECT2 timeout=&quot;1500&quot;  /&gt;
    &lt;pbcast.NAKACK2 use_mcast_xmit=&quot;false&quot; /&gt;
    &lt;UNICAST3 /&gt;
    &lt;pbcast.STABLE desired_avg_gossip=&quot;50000&quot;
                   max_bytes=&quot;4M&quot;/&gt;
    &lt;pbcast.GMS print_local_addr=&quot;true&quot; join_timeout=&quot;2000&quot;/&gt;
    &lt;UFC max_credits=&quot;2M&quot;
         min_threshold=&quot;0.4&quot;/&gt;
    &lt;MFC max_credits=&quot;2M&quot;
         min_threshold=&quot;0.4&quot;/&gt;
    &lt;FRAG2 frag_size=&quot;60K&quot;  /&gt;
&lt;/config&gt;
</code></pre>
<p>Now the command I am testing with:</p>
<pre><code>/var/lib/artemis-instance/bin/artemis perf client --user user --password password --show-latency --url=&quot;tcp://msc-test-hdls-svc:61616?reconnectAttempts=5&amp;confirmationWindowSize=20000&quot; --consumer-url=&quot;tcp://msc-test-hdls-svc:61616?reconnectAttempts=5&quot; queue://performance #--hdr case-1-new.hdr
</code></pre>
<p>Running the command outputs some performance stats and it seems that the client has successfully connected to a broker:</p>
<pre><code>Connection brokerURL = tcp://msc-test-hdls-svc:61616?reconnectAttempts=5&amp;confirmationWindowSize=20000

--- warmup false 
--- sent:           281 msg/sec 
--- blocked:        281 msg/sec 
--- completed:      281 msg/sec 
--- received:       282 msg/sec 
--- send ack time:   mean:   3325.25 us - 50.00%:    679.00 us - 90.00%:   1687.00 us - 99.00%:  81407.00 us - 99.90%:  83455.00 us - 99.99%:  83455.00 us - max:     83455.00 us 
--- transfer time:   mean:   3402.32 us - 50.00%:    787.00 us - 90.00%:   4015.00 us - 99.00%:  78335.00 us - 99.90%:  83455.00 us - 99.99%:  83455.00 us - max:     83455.00 us
</code></pre>
<p>Now when killing a node, I receive the following warning:</p>
<pre><code>2023-06-14 12:56:47,361 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to msc-test-hdls-svc/10.4.1.113:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
</code></pre>
<p>After that stats keep being printed, but are all <code>0 msg/sec</code>.</p>
<p>Then after some more time:</p>
<pre><code>2023-06-14 12:57:52,312 WARN  [org.apache.activemq.artemis.core.client] AMQ212036: Can not find packet to clear: 3 last received command id first stored command id 0
</code></pre>
<p>And the pod running the command is restarted.</p>
<p>My expectation here is that the client will attempt to connect to a different node now after it discovered that one node has been killed.</p>
<p>Either my expectations are wrong or something in my setup isn't working. Any help?</p>
<p>EDIT: I also tried using multiple client IPs in the connection string, but the result was the same.</p>

Moritz Schmitz v. Hülst
<p>The issue here is that you don't have <a href="https://activemq.apache.org/components/artemis/documentation/latest/ha.html" rel="nofollow noreferrer">high availability</a> configured which means that when a broker fails the client will not fail-over to a backup.</p>
<p>In 2.29.0 this will be different as fail-over to another live broker will be supported via <a href="https://issues.apache.org/jira/browse/ARTEMIS-4251" rel="nofollow noreferrer">ARTEMIS-4251</a>.</p>
