<p>I'm setting up a new k8s environment with multiple pods running microservices written in node.js. Several of these services connect to a redis cache.</p>

<p>This is all working most of the time, but I am receiving intermittent errors when accessing redis that make me believe that I'm not connecting correctly, most commonly:</p>

<pre><code>RedisServerException: READONLY You can't write against a read only slave.
</code></pre>

<p>If I try again, I am often successful after two or three attempts.</p>

<p>This is my redis demployment:</p>

<pre>
RESOURCES:

==> v1/Service
NAME                       TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)             AGE
cache-redis-ha-announce-0  ClusterIP  100.xxx.xxx.xxx &lt;none&gt;       6379/TCP,26379/TCP  163m
cache-redis-ha-announce-1  ClusterIP  100.xxx.xxx.xxx &lt;none&gt;       6379/TCP,26379/TCP  163m
cache-redis-ha-announce-2  ClusterIP  100.xxx.xxx.xxx &lt;none&gt;       6379/TCP,26379/TCP  163m
cache-redis-ha             ClusterIP  None            &lt;none&gt;       6379/TCP,26379/TCP  163m

==> v1/StatefulSet
NAME                   DESIRED  CURRENT  AGE
cache-redis-ha-server  3        3        94s

==> v1/Pod(related)
NAME                     READY  STATUS   RESTARTS  AGE
cache-redis-ha-server-0  2/2    Running  0         94s
cache-redis-ha-server-1  2/2    Running  0         64s
cache-redis-ha-server-2  2/2    Running  0         36s

==> v1/ConfigMap
NAME                      DATA  AGE
cache-redis-ha-configmap  3     163m
cache-redis-ha-probes     2     163m


NOTES:
Redis can be accessed via port 6379 and Sentinel can be accessed via port 26379 on the following DNS name from within your cluster:
cache-redis-ha.devtest.svc.cluster.local
</pre>

<p>In my service, I connect to redis thus:</p>

<pre><code>this.client = redis.createClient(6379, "cache-redis-ha.devtest.svc.cluster.local");


this.delAsync = promisify(this.client.del).bind(this.client);


async flush(matchPattern: string): Promise&lt;CacheResult&gt; {
    let result: CacheResult = { matchPattern: matchPattern, reply: true };
    return await this.keysAsync(matchPattern).then(async (keys) =&gt; {
        result.matchedKeys = keys;
        if (keys.length) {
            return await this.delAsync(keys).then((reply) =&gt; {
                result.reply = reply;
                return result;
            });
        }
        return result;
    });
}
</code></pre>

<p>I tried to connecting to the Sentinel port in createClient, but this didn't work.</p>

<p>Is there something obviously wrong in my implementation?</p>

EricP
<p>In a Redis cluster, you have one master which is R/W and several slaves which are RO.</p>
<p>When you use a single service for all Redis pods, your connections will round-robin to all available pods and K8s doesn't know which of them is a master one, that's why you sometimes meet that error. It happens when your connection to a service terminates on RO slave instead of RW master.</p>
<p>You need additional Service and something like a controller or other automation which will point that Service to a right pod which is a master now.</p>
<p>Also, you can get that information from Sentel using <a href="https://redis.io/topics/sentinel#sentinels-and-slaves-auto-discovery" rel="nofollow noreferrer">discovery</a>:</p>
<blockquote>
<p>Sentinels stay connected with other Sentinels in order to reciprocally check the availability of each other, and to exchange messages. However you don't need to configure a list of other Sentinel addresses in every Sentinel instance you run, as Sentinel uses the Redis instances Pub/Sub capabilities in order to discover the other Sentinels that are monitoring the same masters and slaves.</p>
<p>This feature is implemented by sending hello messages into the channel named <strong>sentinel</strong>:hello.</p>
<p>Similarly you don't need to configure what is the list of the slaves attached to a master, as Sentinel will auto discover this list querying Redis.</p>
<p>Every Sentinel publishes a message to every monitored master and slave Pub/Sub channel <strong>sentinel</strong>:hello, every two seconds, announcing its presence with ip, port, runid.</p>
<p>Every Sentinel is subscribed to the Pub/Sub channel <strong>sentinel</strong>:hello of every master and slave, looking for unknown sentinels. When new sentinels are detected, they are added as sentinels of this master.</p>
<p>Hello messages also include the full current configuration of the master. If the receiving Sentinel has a configuration for a given master which is older than the one received, it updates to the new configuration immediately.</p>
<p>Before adding a new sentinel to a master a Sentinel always checks if there is already a sentinel with the same runid or the same address (ip and port pair). In that case all the matching sentinels are removed, and the new added.</p>
</blockquote>
<p>Also, you can <a href="https://redis.io/topics/sentinel#asking-sentinel-about-the-state-of-a-master" rel="nofollow noreferrer">call</a> <code>sentinel master mymaster</code> on any node to get a current master.</p>
<p>So, finally you need to get a Redis Master address (or ID) and use it's Service (in your installation it is <code>cache-redis-ha-announce-*</code>) to connect to a current master.</p>
