<p>I have an issue where I have a workload in Kubernetes which contains an Apache ActiveMQ Artemis broker. The server starts properly when I have a single pod in my workload, the issue starts when I try to scale them. The brokers in the pods can't connect to each other, so I can't scale my work load. My final goal is to make it scalable. I tried it locally with two docker containers and it worked fine.</p>

<p>Here is my <code>broker.xml</code>:</p>

<pre class="lang-xml prettyprint-override"><code>&lt;configuration xmlns="urn:activemq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns:xi="http://www.w3.org/2001/XInclude"
               xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd"&gt;

    &lt;core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="urn:activemq:core "&gt;

        &lt;name&gt;Broker1&lt;/name&gt;

        &lt;broadcast-groups&gt;
            &lt;broadcast-group name="brokerCluster-broadcast"&gt;
                &lt;local-bind-address&gt;0.0.0.0&lt;/local-bind-address&gt;
                &lt;local-bind-port&gt;10000&lt;/local-bind-port&gt;
                &lt;group-address&gt;231.7.7.7&lt;/group-address&gt;
                &lt;group-port&gt;9876&lt;/group-port&gt;
                &lt;broadcast-period&gt;20&lt;/broadcast-period&gt;
                &lt;connector-ref&gt;netty-connector&lt;/connector-ref&gt;
            &lt;/broadcast-group&gt;
        &lt;/broadcast-groups&gt;

        &lt;discovery-groups&gt;
            &lt;discovery-group name="brokerCluster-discovery"&gt;
                &lt;local-bind-port&gt;10000&lt;/local-bind-port&gt;
                &lt;local-bind-address&gt;0.0.0.0&lt;/local-bind-address&gt;
                &lt;group-address&gt;231.7.7.7&lt;/group-address&gt;
                &lt;group-port&gt;9876&lt;/group-port&gt;
                &lt;refresh-timeout&gt;10&lt;/refresh-timeout&gt;
            &lt;/discovery-group&gt;
        &lt;/discovery-groups&gt;

        &lt;cluster-connections&gt;
            &lt;cluster-connection name="brokerCluster"&gt;
                &lt;connector-ref&gt;netty-connector&lt;/connector-ref&gt;
                &lt;retry-interval&gt;500&lt;/retry-interval&gt;
                &lt;use-duplicate-detection&gt;true&lt;/use-duplicate-detection&gt;
                &lt;message-load-balancing&gt;ON_DEMAND&lt;/message-load-balancing&gt;
                &lt;max-hops&gt;1&lt;/max-hops&gt;
                &lt;discovery-group-ref discovery-group-name="brokerCluster-discovery"/&gt;
            &lt;/cluster-connection&gt;
        &lt;/cluster-connections&gt;

        &lt;connectors&gt;
            &lt;connector name="netty-connector"&gt;tcp://0.0.0.0:61610&lt;/connector&gt;
        &lt;/connectors&gt;

        &lt;persistence-enabled&gt;true&lt;/persistence-enabled&gt;

        &lt;journal-type&gt;NIO&lt;/journal-type&gt;

        &lt;paging-directory&gt;data/paging&lt;/paging-directory&gt;

        &lt;bindings-directory&gt;data/bindings&lt;/bindings-directory&gt;

        &lt;journal-directory&gt;data/journal&lt;/journal-directory&gt;

        &lt;large-messages-directory&gt;data/large-messages&lt;/large-messages-directory&gt;

        &lt;journal-datasync&gt;true&lt;/journal-datasync&gt;

        &lt;journal-min-files&gt;2&lt;/journal-min-files&gt;

        &lt;journal-pool-files&gt;10&lt;/journal-pool-files&gt;

        &lt;journal-device-block-size&gt;4096&lt;/journal-device-block-size&gt;

        &lt;journal-file-size&gt;10M&lt;/journal-file-size&gt;

        &lt;journal-buffer-timeout&gt;536000&lt;/journal-buffer-timeout&gt;

        &lt;disk-scan-period&gt;5000&lt;/disk-scan-period&gt;

        &lt;max-disk-usage&gt;90&lt;/max-disk-usage&gt;

        &lt;critical-analyzer&gt;true&lt;/critical-analyzer&gt;

        &lt;critical-analyzer-timeout&gt;120000&lt;/critical-analyzer-timeout&gt;

        &lt;critical-analyzer-check-period&gt;60000&lt;/critical-analyzer-check-period&gt;

        &lt;critical-analyzer-policy&gt;HALT&lt;/critical-analyzer-policy&gt;


        &lt;page-sync-timeout&gt;536000&lt;/page-sync-timeout&gt;

        &lt;acceptors&gt;

            &lt;acceptor name="netty-acceptor"&gt;tcp://0.0.0.0:61610&lt;/acceptor&gt;

            &lt;acceptor name="artemis"&gt;tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true&lt;/acceptor&gt;

            &lt;acceptor name="amqp"&gt;tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true&lt;/acceptor&gt;

            &lt;acceptor name="stomp"&gt;tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true&lt;/acceptor&gt;

            &lt;acceptor name="hornetq"&gt;tcp://0.0.0.0:5445?anycastPrefix=jms.queue.;multicastPrefix=jms.topic.;protocols=HORNETQ,STOMP;useEpoll=true&lt;/acceptor&gt;

            &lt;acceptor name="mqtt"&gt;tcp://0.0.0.0:1883?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=MQTT;useEpoll=true&lt;/acceptor&gt;

        &lt;/acceptors&gt;


        &lt;security-settings&gt;
            &lt;security-setting match="#"&gt;
                &lt;permission type="createNonDurableQueue" roles="amq"/&gt;
                &lt;permission type="deleteNonDurableQueue" roles="amq"/&gt;
                &lt;permission type="createDurableQueue" roles="amq"/&gt;
                &lt;permission type="deleteDurableQueue" roles="amq"/&gt;
                &lt;permission type="createAddress" roles="amq"/&gt;
                &lt;permission type="deleteAddress" roles="amq"/&gt;
                &lt;permission type="consume" roles="amq"/&gt;
                &lt;permission type="browse" roles="amq"/&gt;
                &lt;permission type="send" roles="amq"/&gt;
                &lt;permission type="manage" roles="amq"/&gt;
            &lt;/security-setting&gt;
        &lt;/security-settings&gt;

        &lt;address-settings&gt;
            &lt;address-setting match="activemq.management#"&gt;
                &lt;dead-letter-address&gt;DLQ&lt;/dead-letter-address&gt;
                &lt;expiry-address&gt;ExpiryQueue&lt;/expiry-address&gt;
                &lt;redelivery-delay&gt;0&lt;/redelivery-delay&gt;
                &lt;max-size-bytes&gt;-1&lt;/max-size-bytes&gt;
                &lt;message-counter-history-day-limit&gt;10&lt;/message-counter-history-day-limit&gt;
                &lt;address-full-policy&gt;PAGE&lt;/address-full-policy&gt;
                &lt;auto-create-queues&gt;true&lt;/auto-create-queues&gt;
                &lt;auto-create-addresses&gt;true&lt;/auto-create-addresses&gt;
                &lt;auto-create-jms-queues&gt;true&lt;/auto-create-jms-queues&gt;
                &lt;auto-create-jms-topics&gt;true&lt;/auto-create-jms-topics&gt;
            &lt;/address-setting&gt;
            &lt;address-setting match="#"&gt;
                &lt;dead-letter-address&gt;DLQ&lt;/dead-letter-address&gt;
                &lt;expiry-address&gt;ExpiryQueue&lt;/expiry-address&gt;
                &lt;redelivery-delay&gt;0&lt;/redelivery-delay&gt;
                &lt;redistribution-delay&gt;0&lt;/redistribution-delay&gt;
                &lt;max-size-bytes&gt;-1&lt;/max-size-bytes&gt;
                &lt;message-counter-history-day-limit&gt;10&lt;/message-counter-history-day-limit&gt;
                &lt;address-full-policy&gt;PAGE&lt;/address-full-policy&gt;
                &lt;auto-create-queues&gt;true&lt;/auto-create-queues&gt;
                &lt;auto-create-addresses&gt;true&lt;/auto-create-addresses&gt;
                &lt;auto-create-jms-queues&gt;true&lt;/auto-create-jms-queues&gt;
                &lt;auto-create-jms-topics&gt;true&lt;/auto-create-jms-topics&gt;
            &lt;/address-setting&gt;
        &lt;/address-settings&gt;

        &lt;addresses&gt;
            &lt;address name="DLQ"&gt;
                &lt;anycast&gt;
                    &lt;queue name="DLQ" /&gt;
                &lt;/anycast&gt;
            &lt;/address&gt;
            &lt;address name="ExpiryQueue"&gt;
                &lt;anycast&gt;
                    &lt;queue name="ExpiryQueue" /&gt;
                &lt;/anycast&gt;
            &lt;/address&gt;
            &lt;address name="TestQueue"&gt;
                &lt;anycast&gt;
                    &lt;queue name="testQueue" /&gt;
                &lt;/anycast&gt;
            &lt;/address&gt;
        &lt;/addresses&gt;
    &lt;/core&gt;
&lt;/configuration&gt;
</code></pre>

<p>Edit: Attached kubernetes,docker configs</p>

<p><strong>deployment.yml</strong></p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: artemis
  labels:
    app: artemis
spec:
  ports:
  - port: 6161
    name: service
    protocol: UDP
  - port: 8161
    name: console
    protocol: UDP
  - port: 9876
    name: broadcast
    protocol: UDP
  - port: 61610
    name: netty-connector
    protocol: TCP
  - port: 5672
    name: acceptor-amqp
    protocol: TCP
  - port: 61613
    name: acceptor-stomp
    protocol: TCP
  - port: 5445
    name: accep-hornetq
    protocol: TCP
  - port: 1883
    name: acceptor-mqt
    protocol: TCP
  - port: 10000
    protocol: UDP
    name: brokercluster-broadcast // this name is invalid but i wanted to match it to my broker.xml
  clusterIP: None
  selector:
    app: artemis01
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: artemis01headless
  namespace: artemis
spec:
  selector:
    matchLabels:
      app: artemis01 
  serviceName: artemis01
  replicas: 3
  template:
    metadata:
      labels:
        app: artemis01 
    spec:
      affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: app
                  operator: In
                  values:
                  - worker
      containers:
        - env:
          - name: ARTEMIS_PASSWORD
            value: admin
          - name: ARTEMIS_USER
            value: admin
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
          name: artemis
          image: 
          ports:
          - containerPort: 6161
            name: service
            protocol: UDP
          - containerPort: 8161
            name: console
            protocol: UDP
          - containerPort: 9876
            name: broadcast
            protocol: UDP
          - containerPort: 61610
            name: netty-connector
            protocol: TCP
          - containerPort: 5672
            name: acceptor-amqp
            protocol: TCP
          - containerPort: 61613
            name: acceptor-stomp
            protocol: TCP
          - containerPort: 5445
            name: accep-hornetq
            protocol: TCP
          - containerPort: 1883
            name: acceptor-mqtt
            protocol: TCP
          - containerPort: 10000
            name: brokercluster-broadcast
            protocol: UDP
      imagePullSecrets:
        - name: xxxxxxx

</code></pre>

<p><a href="https://github.com/apache/activemq-artemis/tree/master/artemis-docker" rel="nofollow noreferrer"><strong>Dockerfile</strong> source</a> </p>

<pre><code># Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

# ActiveMQ Artemis

FROM jboss/base-jdk:8
LABEL maintainer="Apache ActiveMQ Team"
# Make sure pipes are considered to determine success, see: https://github.com/hadolint/hadolint/wiki/DL4006
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
WORKDIR /opt

ENV ARTEMIS_USER artemis
ENV ARTEMIS_PASSWORD artemis
ENV ANONYMOUS_LOGIN false
ENV CREATE_ARGUMENTS --user ${ARTEMIS_USER} --password ${ARTEMIS_PASSWORD} --silent --http-host 0.0.0.0 --relax-jolokia

USER root

# add user and group for artemis
RUN groupadd -g 1001 -r artemis &amp;&amp; useradd -r -u 1001 -g artemis artemis \
 &amp;&amp; yum install -y libaio &amp;&amp; yum -y clean all

USER artemis

ADD . /opt/activemq-artemis

# Web Server
EXPOSE 8161 \
    61610 \
    9876 \
    61613 \
    61616 \
    5672 \
    5445 \
    1883 \
    10000

USER root

RUN mkdir /var/lib/artemis-instance &amp;&amp; chown -R artemis.artemis /var/lib/artemis-instance

COPY ./docker/docker-run.sh /

USER artemis

# Expose some outstanding folders
VOLUME ["/var/lib/artemis-instance"]
WORKDIR /var/lib/artemis-instance

ENTRYPOINT ["/docker-run.sh"]
CMD ["run"]

</code></pre>

<p><strong>run.sh</strong></p>

<pre class="lang-sh prettyprint-override"><code>set -e

BROKER_HOME=/var/lib/
CONFIG_PATH=$BROKER_HOME/etc
export BROKER_HOME OVERRIDE_PATH CONFIG_PATH

echo CREATE_ARGUMENTS=${CREATE_ARGUMENTS}

if ! [ -f ./etc/broker.xml ]; then
    /opt/activemq-artemis/bin/artemis create ${CREATE_ARGUMENTS} .
    #the script copies my broker.xml to /var/lib/artemis-instance/etc/broker.xml here.
    sed -i -e 's|$PLACEHOLDERIP|'$MY_POD_IP'|g' /var/lib/artemis-instance/etc/broker.xml
else
    echo "broker already created, ignoring creation"
fi

exec ./bin/artemis "$@"
</code></pre>

Bence
<p>I believe the issue is with your <code>connector</code> configuration. This is what you're using:</p>

<pre class="lang-xml prettyprint-override"><code>&lt;connector name="netty-connector"&gt;tcp://0.0.0.0:61610&lt;/connector&gt;
</code></pre>

<p>The information from this connector gets broadcast to the other cluster members since you've specified it in the <code>&lt;connector-ref&gt;</code> of your <code>&lt;cluster-connection&gt;</code>. The other cluster members then try to use this information to connect back to the node that broadcast it. However, 0.0.0.0 won't make sense to a remote client.</p>

<p>The address 0.0.0.0 is a meta-address. In the context of a listener (e.g. an Artemis <code>acceptor</code>) it means that the listener will listen for connections on all local addresses. In the context of a <code>connector</code> it doesn't really have a meaning. See <a href="http://en.wikipedia.org/wiki/0.0.0.0" rel="nofollow noreferrer">this article for more about 0.0.0.0</a>.</p>

<p>You should be using a <em>real</em> IP address or hostname that a client can use to actually get a network route to the server.</p>

<p>Also, since you're using UDP multicast (i.e. via the <code>&lt;broadcast-group&gt;</code> and <code>&lt;discovery-group&gt;</code>) please ensure this functions as well between the containers/pods. If you can't get UDP multicast working in your environment (or simply don't want to) you could switch to a static cluster configuration. Refer to <a href="https://activemq.apache.org/components/artemis/documentation/latest/clusters.html#discovery-using-static-connectors" rel="nofollow noreferrer">the documentation</a> and <a href="https://github.com/apache/activemq-artemis/tree/master/examples/features/clustered/clustered-static-discovery" rel="nofollow noreferrer">the "clustered static discovery" example</a> for details on how to configure this.</p>
