<p>I'm running locust <code>locust==2.8.6</code> on python 3.10. I run it on kubernetes via AWS EKS. I run it distributed and am trying to set up 1 master and 5 workers.</p>
<p>The master pod starts with command:</p>
<pre><code>command: [&quot;locust&quot;]
        args: [&quot;-f&quot;,&quot;$filename&quot;,&quot;--headless&quot;,&quot;--users=$clients&quot;,&quot;--spawn-rate=$hatch-rate&quot;,&quot;--run-time=$run-time&quot;,&quot;--only-summary&quot;,&quot;--master&quot;,&quot;--expect-workers=$num_slaves&quot;]
</code></pre>
<p>and the workers start with command:</p>
<pre><code>command: [&quot;locust&quot;]
        args: [&quot;-f&quot;,&quot;$filename&quot;,&quot;--worker&quot;,&quot;--master-host=locust-master$task_id&quot;]
</code></pre>
<p>Indeed, on a worker pod, I can run <code>telnet locust-master1 5557</code> and confirm communication. (In that case, $task_id=1)</p>
<p>I see logs in the master pod as below:</p>
<pre><code>[2022-04-27 22:53:16,969] locust-master1--1-z2lr8/INFO/root: Waiting for workers to be ready, 0 of 5 connected
[2022-04-27 22:53:17,109] locust-master1--1-z2lr8/INFO/locust.runners: Client 'locust-slave1-tt7n5_fec1320a406b42319f3088bd9a7c181c' reported as ready. Currently 1 clients ready to swarm.
[2022-04-27 22:53:17,147] locust-master1--1-z2lr8/INFO/locust.runners: Client 'locust-slave1-qv7kt_011dbeb9f15d452f935c5643fb463632' reported as ready. Currently 2 clients ready to swarm.
[2022-04-27 22:53:17,261] locust-master1--1-z2lr8/INFO/locust.runners: Client 'locust-slave1-ks5wb_356fcf54ac2644e4badc684e3846520c' reported as ready. Currently 3 clients ready to swarm.
[2022-04-27 22:53:17,354] locust-master1--1-z2lr8/INFO/locust.runners: Client 'locust-slave1-cbkbd_2c90cedde5224e1e9cf47bbb543b9097' reported as ready. Currently 4 clients ready to swarm.
[2022-04-27 22:53:17,364] locust-master1--1-z2lr8/INFO/locust.runners: Client 'locust-slave1-xfvsz_196bba3928c5491e896acd411798d48d' reported as ready. Currently 5 clients ready to swarm.
[2022-04-27 22:53:17,970] locust-master1--1-z2lr8/INFO/locust.main: Run time limit set to 5400 seconds
[2022-04-27 22:53:17,971] locust-master1--1-z2lr8/INFO/locust.main: Starting Locust 2.8.6
[2022-04-27 22:53:17,971] locust-master1--1-z2lr8/INFO/locust.runners: Sending spawn jobs of 50 users at 0.50 spawn rate to 5 ready clients
[2022-04-27 22:53:17,977] locust-master1--1-z2lr8/INFO/locust_submit_judgments: Locust Startup: job_id: 1434194
[2022-04-27 22:53:18,376] locust-master1--1-z2lr8/INFO/locust.runners: Worker locust-slave1-cbkbd_2c90cedde5224e1e9cf47bbb543b9097 failed to send heartbeat, setting state to missing.
[2022-04-27 22:53:20,384] locust-master1--1-z2lr8/INFO/locust.runners: Worker locust-slave1-qv7kt_011dbeb9f15d452f935c5643fb463632 failed to send heartbeat, setting state to missing.
[2022-04-27 22:53:20,385] locust-master1--1-z2lr8/INFO/locust.runners: Worker locust-slave1-ks5wb_356fcf54ac2644e4badc684e3846520c failed to send heartbeat, setting state to missing.
[2022-04-27 22:53:22,391] locust-master1--1-z2lr8/INFO/locust.runners: Worker locust-slave1-tt7n5_fec1320a406b42319f3088bd9a7c181c failed to send heartbeat, setting state to missing.
[2022-04-27 22:53:22,391] locust-master1--1-z2lr8/INFO/locust.runners: Worker locust-slave1-xfvsz_196bba3928c5491e896acd411798d48d failed to send heartbeat, setting state to missing.
[2022-04-27 22:53:22,392] locust-master1--1-z2lr8/INFO/locust.runners: The last worker went missing, stopping test.
[2022-04-27 22:53:22,392] locust-master1--1-z2lr8/INFO/locust_submit_judgments: Locust Teardown: sending query messages to Results DB
</code></pre>
<p>So I do see the workers register themselves, but as soon as the test starts, the master pod says the workers fail to send heartbeats and sets them to missing. If I run the master pod without <code>--headless</code>, that means I can open the web UI and manually start the job. I see the same problem: when I start the job manually, the same heartbeat messages appear.</p>
<p>On the worker pods, I see my debug startup logs and nothing that indicates an issue.</p>
<p>I cannot find a guide online on how to set up distributed locust (except for when it was called locustio and in version 0.x) and things have changed a lot since then.</p>
<p>What needs to be set here? I am not sure which code to include without including many lines of setup code. I am trying to test against postgres so I'm thinking of following <a href="https://docs.locust.io/en/stable/testing-other-systems.html" rel="nofollow noreferrer">https://docs.locust.io/en/stable/testing-other-systems.html</a> but in all the examples they are wrapping attributes which is a departure from the code I've inherited.</p>

swagrov
<p>have you checked CPU utilization? We had a similar case when VM's were having 100 CPU consumption and worker simply didn't had the possibility to send heartbeat.</p>
