<p>I am trying to configure a Redis cluster onto Kubernetes with an Istio Mesh installed. The Redis Cluster is able to be created without Istio and each Pods are auto-injected with an Istio Proxy (Envoy). However, with Istio installed and the Istio proxy attached to each Redis Pods, the Redis cluster is not able to "meet" correctly through the CLUSTER MEET command from the CLI.</p>

<p>For instance, I have Redis Pod A (slot 0 - 10919) and Redis Pod B (slot 10920 - 16383). This is the result after attempting a CLUSTER MEET command between them (cluster meet ClusterIPForRedisPodB 6379). </p>

<p>For Redis Pod A, the cluster info is updated and includes Redis Pod B:</p>

<p><a href="https://i.stack.imgur.com/OkRU4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OkRU4.png" alt="Redis Pod A"></a></p>

<p>On the contrary, for Redis Pod B, the cluster info is not updated and does not include Redis Pod A:</p>

<p><a href="https://i.stack.imgur.com/UYnp0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/UYnp0.png" alt="Redis Pod B"></a></p>

<p>I am able to send curl and netcat responses between the two Pods for Port 16379 and 6379. In addition, Envoy appears to have these ports opened as well. </p>

Kevin Chow
<p>I have replicated your issue and found a solution to your problem.</p>

<p>Let me start by explaining what was the cause of your problem.</p>

<p>Redis gossip protocol works this way:
when you type <code>cluster meet &lt;ip&gt; &lt;port&gt;</code> on <em>redis1</em>,
<em>redis1</em> opens a tcp connection to <em>redis2</em>.
In normal case, when <em>redis2</em> receives a connection, it accepts it, looks up source ip address of who's connecting
and also opens a tcp connection to that address, so to <em>redis1</em> in that case.
(More on how gossip protocol works inredis can be found in <a href="https://redis.io/topics/cluster-spec" rel="nofollow noreferrer">redis documentation</a>,
or in <a href="https://medium.com/@Alibaba_Cloud/in-depth-analysis-of-redis-cluster-gossip-protocol-344b01f71c03" rel="nofollow noreferrer">this article</a>)</p>

<p>Here comes the <em>istio</em> part.
Istio by default configures envoy as a typical proxy and as you can read in
<a href="https://istio.io/docs/reference/config/istio.mesh.v1alpha1/#ProxyConfig-InboundInterceptionMode" rel="nofollow noreferrer">istio documentation</a>:<br><br>
<a href="https://i.stack.imgur.com/AwfJB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AwfJB.png" alt="interception mode"></a></p>

<p>Istio be default is using <code>REDIRECT</code> proxing and as states in documentation:</p>

<blockquote>
  <p>This mode loses source IP addresses during redirection</p>
</blockquote>

<p>This is our source of problem.</p>

<p><em>redis2</em> when receives a connection, it sees it as coming from localhost. Envoy has lost source IP address of <em>redis1</em> and <em>redis2</em> is now unable to
open a connection back to <em>redis1</em>.</p>

<p>Now, we have some options:</p>

<ol>
<li>you can try to change proxing mode to <code>TPROXY</code> (I tried it but couldn't make it work)</li>
<li>use redis built-in config variable</li>
</ol>

<p>Lets take a look at the second option a bit closer because this is the one that worked for me.
In <a href="https://raw.githubusercontent.com/antirez/redis/5.0/redis.conf" rel="nofollow noreferrer"><code>redis.conf</code></a>
file you can find this section:</p>

<blockquote>
  <h3>CLUSTER DOCKER/NAT support</h3>
  
  <p>In certain deployments, Redis Cluster nodes address discovery fails, because
  addresses are NAT-ted or because ports are forwarded (the typical case is
  Docker and other containers).</p>
  
  <p>In order to make Redis Cluster working in such environments, a static
  configuration where each node knows its public address is needed. The
  following two options are used for this scope, and are:</p>
  
  <ul>
  <li>cluster-announce-ip</li>
  <li>cluster-announce-port</li>
  <li>cluster-announce-bus-port</li>
  </ul>
  
  <p>Each instruct the node about its address, client port, and cluster message
  bus port. The information is then published in the header of the bus packets
  so that other nodes will be able to correctly map the address of the node
  publishing the information.</p>
  
  <p>If the above options are not used, the normal Redis Cluster auto-detection
  will be used instead.</p>
  
  <p>Note that when remapped, the bus port may not be at the fixed offset of
  clients port + 10000, so you can specify any port and bus-port depending
  on how they get remapped. If the bus-port is not set, a fixed offset of
  10000 will be used as usually.</p>
  
  <p>Example:</p>
  
  <p>cluster-announce-ip 10.1.1.5<br>
  cluster-announce-port 6379<br>
  cluster-announce-bus-port 6380<br></p>
</blockquote>

<p><strong>We need to set <code>cluster-announce-ip</code> variable to redis's pod own ip address.</strong></p>

<p>You can do it for example modifying <code>redis-cluster</code> ConfigMap like this
(It's modified redis configmap from <a href="https://rancher.com/blog/2019/deploying-redis-cluster/" rel="nofollow noreferrer">this article</a>):</p>

<pre><code>---
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster
data:
  update-node.sh: |
      #!/bin/sh
      REDIS_NODES="/data/nodes.conf"
      sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES}
      cp /conf/redis.conf /redis.conf # &lt;------HERE-----
      sed -i "s/MY_IP/${POD_IP}/" /redis.conf # &lt;------HERE-----
      exec "$@"
  redis.conf: |+
      cluster-enabled yes
      cluster-require-full-coverage no
      cluster-node-timeout 15000
      cluster-config-file /data/nodes.conf
      cluster-migration-barrier 1
      appendonly yes
      protected-mode no
      cluster-announce-ip MY_IP # &lt;------HERE-----
</code></pre>

<p>Also remember to change you container's <code>command</code> like this to point to the right <code>redis.conf</code> file:</p>

<pre><code>command: ["/conf/update-node.sh", "redis-server", "/redis.conf"]
</code></pre>

<p>Every redis node will now advertise this address as it's own so the other redis nodes will now know how to connect with it.</p>

<p>Let me know if it helped.</p>
