<p>My architecture: namespaces in kubernetes behind proxy, firewall and other switches.
My master sends tasks to the slaves and when the script ends, it closes the slaves and the master. I have netpols, ingress and egress configured and running. I have open ports in the 3-port ranges, e.g. 60000-60002, 1099-1101.</p>
<p>My problem: Normally, when I start a test, the master should send logs such as :</p>
<pre><code>&quot;Oct 12, 2022 11:31:39 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Creating summariser &lt;summary&gt;
Created the tree successfully using /mount/10_2022/scenario.jmx
Configuring remote engine: slave-1
Using local port: 60000
Starting distributed test with remote engines: [slave-1] @ October 12, 2022 11:31:41 AM CEST (1665567101036)
Remote engines have been started:[opl-jmeter-distributed-slave-1]
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445&quot;
</code></pre>
<p>While the master sends logs :</p>
<pre><code>&quot;Oct 12, 2022 11:31:39 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Creating summariser &lt;summary&gt;
Created the tree successfully using /mount/10_2022/scenario.jmx
Configuring remote engine: slave-1
Using local port: 60000
Starting distributed test with remote engines: [slave-1] @ October 12, 2022 11:31:41 AM CEST (1665567101036)
</code></pre>
<p><strong>&lt;Between this lines is about 7-10 minutes break.&gt;</strong></p>
<pre><code>&quot;Remote engines have been started:[opl-jmeter-distributed-slave-1]
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445&quot;
</code></pre>
<p>nothing happens after that, except log errors like :</p>
<pre><code>&quot;2022-10-12 11:40:30,274 ERROR o.a.j.s.RemoteListenerWrapper: testStarted(host) on slave-1
java.rmi.ConnectException: Connection refused to host: &lt;MASTER'S IP&gt;; nested exception is: java.net.ConnectException: Connection timed out (Connection timed out)&quot;
</code></pre>
<p>and</p>
<pre><code>&quot;2022-10-12 11:49:18,668 ERROR o.a.j.s.BatchSampleSender: sampleOccurred
java.rmi.ConnectException: Connection refused to host: &lt;MASTER'S IP&gt;; nested exception is:java.net.ConnectException: Connection timed out (Connection timed out)&quot;
</code></pre>
<p>I started debugging my actions and this is what I get from the master logs:</p>
<pre><code>&quot;Oct 13, 2022 10:45:37 AM sun.rmi.transport.tcp.TCPChannel free
FINE: RMI RenewClean-[172.16.28.15:60000,SSLRMIClientSocketFactory(keyStoreLocation=/mount/keyStore/rmi_keystore.jks, type=JKS, trustStoreLocation=/mount/keyStore/rmi_keystore.jks, type=JKS, alias=rmi)]: create reaper
Oct 13, 2022 10:45:52 AM sun.rmi.transport.tcp.TCPChannel$1 run
FINER: RMI Scheduler(0): wake up
Oct 13, 2022 10:45:52 AM sun.rmi.transport.tcp.TCPChannel freeCachedConnections
FINER: RMI Scheduler(0): connection timeout expired
Oct 13, 2022 10:45:52 AM sun.rmi.transport.tcp.TCPConnection close
FINE: RMI Scheduler(0): close connection&quot;
</code></pre>
<p>I started debugging my actions and this is what I get from the master logs:</p>
<pre><code>Oct 13, 2022 10:45:37 AM sun.rmi.transport.tcp.TCPChannel free
FINE: RMI RenewClean-[172.16.28.15:60000,SSLRMIClientSocketFactory(keyStoreLocation=/mount/keyStore/rmi_keystore.jks, type=JKS, trustStoreLocation=/mount/keyStore/rmi_keystore.jks, type=JKS, alias=rmi)]: create reaper
Oct 13, 2022 10:45:52 AM sun.rmi.transport.tcp.TCPChannel$1 run
FINER: RMI Scheduler(0): wake up
Oct 13, 2022 10:45:52 AM sun.rmi.transport.tcp.TCPChannel freeCachedConnections
FINER: RMI Scheduler(0): connection timeout expired
Oct 13, 2022 10:45:52 AM sun.rmi.transport.tcp.TCPConnection close
FINE: RMI Scheduler(0): close connection
</code></pre>
<p>Please help. Any clue? Why my test doesn't start?</p>
<p><strong>--- UPDATE ---</strong></p>
<p><em><strong>My slave yaml:</strong></em></p>
<pre><code>&gt; apiVersion: v1
kind: Service
metadata:
  name: jmeter-distributed-slave-1
  labels:
    run: jmeter-slave
  namespace: tst-eff
spec:
  type: ClusterIP
  ports:
  - port: 8000
    name: debug
  - port: 60000
    name: input
  - port: 1099
    name: port
  selector:
    id: jmeter-distributed-slave-1
---
apiVersion: v1
kind: Pod
metadata:
  name: jmeter-distributed-slave-1
  labels:
    run: jmeter-slave
    layer: slave
    type: distributed
    id: jmeter-distributed-slave-1
  namespace: tst-eff
spec:
  restartPolicy: Never
  containers:
  - name: jmeter-distributed-slave-1
    image: performance/jmeter:5.5
    imagePullPolicy: Always
    env:
    - name: JVM_ARGS
      value: &quot;-server -d64 -Xms12g -Xmx31g -Xss256k -XX:+UseConcMarkSweepGC -XX:NewSize=512m -XX:MaxNewSize=1024m -XX:+DisableExplicitGC&quot;
    ports:
    - containerPort: 60000
    - containerPort: 8000
    volumeMounts:
    - mountPath: &quot;/mount&quot;
      name: properties
    - mountPath: &quot;/users&quot;
      name: users
    command: [&quot;jmeter.sh&quot;]
    args: [&quot;-s&quot;, &quot;-Jserver.rmi.localport=60000&quot;, &quot;-Jserver.rmi.ssl.keystore.file=/mount/keyStore/rmi_keystore.jks&quot;, &quot;-Jhttpclient4.retrycount=1&quot;, &quot;-Jhttpclient4.idletimeout=5000&quot;, &quot;-Jhttp.connection.stalecheck$Boolean=true&quot;, &quot;-q&quot;, &quot;/mount/prod.properties&quot;, &quot;-D&quot;, &quot;javax.net.ssl.keyStore=/mount/keyStore/keystore_prod.jks&quot;, &quot;-D&quot;, &quot;javax.net.ssl.keyStorePassword=passwordpassword&quot;, &quot;-j&quot;, &quot;/mount/logs_jmeter/jmeter-distributed-slave-1_'YYYYMMdd_HHmm'.log&quot;]
    resources:
      requests:
        cpu: 2
        memory: 12G
      limits:
        cpu: 3
        memory: 30G
  imagePullSecrets:
  - name: credentials
  securityContext:
    fsGroup: xxx
  volumes:
  - name: properties    
    nfs:
      server: xxx.xx.xx.xx
      path: /performance_tst_eff/jmeter/
  - name: users
    nfs:
      server: yyy.yy.yy.yy
      path: /performance_tst_eff/jmeter-1-r1
</code></pre>
<p><em><strong>My master yaml:</strong></em></p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: jmeter-distributed-master
  labels:
    run: jmeter-master
  namespace: tst-eff
spec:
  type: ClusterIP
  ports:
  - port: 8000
    name: debug
  - port: 60000
    name: input
  - port: 1099
    name: port
  selector:
    id: jmeter-distributed-master
---
apiVersion: v1
kind: Pod
metadata:
  name: jmeter-distributed-master
  labels:
    run: jmeter-master
    type: distributed
    layer: master
    id: jmeter-distributed-master  
  namespace: tst-eff
spec:
  restartPolicy: Never
  containers:
  - name: jmeter-distributed-master
    image: performance/jmeter:5.5
    imagePullPolicy: Always
    env:
    - name: UMASK
      value: &quot;011&quot;
    - name: JVM_ARGS
      value: &quot;-server -d64 -Xms1g -Xmx2g -Xss256k -XX:+UseConcMarkSweepGC -XX:NewSize=512m -XX:MaxNewSize=1024m -XX:+DisableExplicitGC&quot;
    ports:
    - containerPort: 60000
    volumeMounts:
    - mountPath: &quot;/mount&quot;
      name: properties
    - mountPath: &quot;/users&quot;
      name: users
    command: [&quot;bash&quot;]
    resources:
      requests:
        cpu: 1
        memory: 4G
      limits:
        cpu: 4
        memory: 8G
    args: [&quot;-c&quot;, &quot;/jmeter/bin/jmeter.sh -n -Jserver.rmi.localport=60000 -Jserver.rmi.ssl.keystore.file=/mount/keyStore/rmi_keystore.jks -Jclient.tries=3 -Jclient.retries_delay=10000 -Ghttpclient4.retrycount=1 -Ghttpclient4.idletimeout=5000 -Ghttp.connection.stalecheck$Boolean=true -D javax.net.ssl.keyStore=/mount/keyStore/keystore_prod.jks -D javax.net.ssl.keyStorePassword=passwordpassword -q /mount/prod.properties -l /mount/results/mini_smoke_$(date +%Y%m%d%H%M).jtl -t /mount/10_2022/mini.jmx -j /mount/logs_jmeter/jmeter-distributed-master_$(date +%Y%m%d%H%M).log -R jmeter-distributed-slave-1 -X&quot;]
  imagePullSecrets:
  - name: credentials
  securityContext:
    fsGroup: xxxx
  volumes:
  - name: properties
    nfs:
      server: xxx.xx.xx.xx
      path: /performance_tst_eff/jmeter/
  - name: users
    nfs:
      server: yyy.yy.yy.yy
      path: /performance_tst_eff/jmeter-1-r1  
</code></pre>
<p>Where is wrong?
The following ports I have open on the firewall:
1099-1101
8000-8002
60000-60002</p>

Szpindel Szpindel
<p>Ok, I figured it out. My firewall did not allow connection between slave and master on <a href="https://en.wikipedia.org/wiki/Ephemeral_port" rel="nofollow noreferrer">ephemeral</a> ports. The slave wants to communicate with the master on ephemeral ports and when I opened those ports, everything starts working.</p>
