<p>I have two nameko services that communicate using RPC via RabbitMQ. Locally with docker-compose all works fine. Then I deployed everything to Kubernetes/Istio cluster on DigitalOcean and started get the following errors. It repeats continuously 1 time in 10/20/60 minutes. Communication between services works fine (before and after recconect I suppose) but logs are messy with those unexpected reconnections that should not happen.</p>

<p><a href="https://www.dropbox.com/s/5vlevrzdahb67h1/rabbit.yaml?dl=0" rel="nofollow noreferrer">Helm RabbitMQ configuration file</a></p>

<p>I tried to increase RAM and CPU configuration (to the values in the configuration files above: 512Mb and 400m) but still have the same behavior.</p>

<p><strong>NB: I don't touch services after deployment, no messages being sent or any requests made and I have this error for the first time in around 60 minutes.</strong> When I make requests they succeed but eventually we still have this errors in logs afterwards.</p>

<p><strong>Nameko service log:</strong></p>

<pre><code>"Connection to broker lost, trying to re-establish connection...",
"exc_info": "Traceback (most recent call last):
File \"/usr/local/lib/python3.6/site-packages/kombu/mixins.py\", line 175, in run for _ in self.consume(limit=None, **kwargs):
File \"/usr/local/lib/python3.6/site-packages/kombu/mixins.py\", line 197, in consume   conn.drain_events(timeout=safety_interval)
File \"/usr/local/lib/python3.6/site-packages/kombu/connection.py\", line 323, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File \"/usr/local/lib/python3.6/site-packages/kombu/transport/pyamqp.py\", line 103, in drain_events
return connection.drain_events(**kwargs)
File \"/usr/local/lib/python3.6/site-packages/amqp/connection.py\", line 505, in drain_events
while not self.blocking_read(timeout):
File \"/usr/local/lib/python3.6/site-packages/amqp/connection.py\", line 510, in blocking_read\n    frame = self.transport.read_frame()
File \"/usr/local/lib/python3.6/site-packages/amqp/transport.py\", line 252, in read_frame
frame_header = read(7, True)
File \"/usr/local/lib/python3.6/site-packages/amqp/transport.py\", line 446, in _read
raise IOError('Server unexpectedly closed connection')
OSError: Server unexpectedly closed connection"}
{"name": "kombu.mixins", "asctime": "29/12/2019 20:22:54", "levelname": "INFO", "message": "Connected to amqp://user:**@rabbit-rabbitmq:5672//"}
</code></pre>

<p><strong>RabbitMQ log</strong></p>

<pre><code>2019-12-29 20:22:54.563 [warning] &lt;0.718.0&gt; closing AMQP connection &lt;0.718.0&gt; (127.0.0.1:46504 -&gt; 127.0.0.1:5672, vhost: '/', user: 'user'):
client unexpectedly closed TCP connection
2019-12-29 20:22:54.563 [warning] &lt;0.705.0&gt; closing AMQP connection &lt;0.705.0&gt; (127.0.0.1:46502 -&gt; 127.0.0.1:5672, vhost: '/', user: 'user'):
client unexpectedly closed TCP connection
2019-12-29 20:22:54.681 [info] &lt;0.3424.0&gt; accepting AMQP connection &lt;0.3424.0&gt; (127.0.0.1:43466 -&gt; 127.0.0.1:5672)
2019-12-29 20:22:54.689 [info] &lt;0.3424.0&gt; connection &lt;0.3424.0&gt; (127.0.0.1:43466 -&gt; 127.0.0.1:5672): user 'user' authenticated and granted access to vhost '/'
2019-12-29 20:22:54.690 [info] &lt;0.3431.0&gt; accepting AMQP connection &lt;0.3431.0&gt; (127.0.0.1:43468 -&gt; 127.0.0.1:5672)
2019-12-29 20:22:54.696 [info] &lt;0.3431.0&gt; connection &lt;0.3431.0&gt; (127.0.0.1:43468 -&gt; 127.0.0.1:5672): user 'user' authenticated and granted access to vhost '/'

</code></pre>

<p><strong>UPD:</strong></p>

<p><a href="https://www.dropbox.com/s/y1nfuv844jm6z5p/rabbit-pod.yaml?dl=0" rel="nofollow noreferrer">Rabbit pod yaml</a></p>

<p>Issue is with istio proxy getting injected as sidecar container inside rabbitmq pod. You need to exclude istio proxy from rabbitmq then it should work. </p>
