<p>I am currently switching from Service Fabric to Kubernetes and was wondering how to do custom and more complex load balancing.</p>

<p>So far I already read about Kubernetes offering "Services" which do load balancing for pods hidden behind them, but this is only available in more plain ways.</p>

<p>What I want to rewrite right now looks like the following in Service Fabric:</p>

<p>I have this interface:  </p>

<pre><code>public interface IEndpointSelector
{
    int HashableIdentifier { get; }
}
</code></pre>

<p>A context keeping track of the account in my ASP.Net application e.g. inherits this. Then, I wrote some code which would as of now do service discovery through the service fabric cluster API and keep track of all services, updating them when any instances die or are being respawned.</p>

<p>Then, based on the deterministic nature of this identifier (due to the context being cached etc.) and given multiple replicas of the target service of a frontend -> backend call, I can reliably route traffic for a certain account to a certain endpoint instance.</p>

<p>Now, how would I go about doing this in Kubernetes?</p>

<p>As I already mentioned, I found "Services", but it seems like their load balancing does not support custom logic and is rather only useful when working with stateless instances.</p>

<p>Is there also a way to have service discovery in Kubernetes which I could use here to replace my existing code at some points?</p>

Sossenbinder
<h1>StatefulSet</h1>

<p><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/" rel="noreferrer">StatefulSet</a> is a <em>building block</em> for stateful workload on Kubernetes with certain guarantees.</p>

<h2>Stable and unique network identity</h2>

<blockquote>
  <p>StatefulSet Pods have a unique identity that is comprised of an ordinal, a stable network identity, and stable storage.</p>
</blockquote>

<p>As an example, if your StatefulSet has the name <code>sharded-svc</code></p>

<pre><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sharded-svc
</code></pre>

<p>And you have e.g. 3 replicas, those will be named by <code>&lt;name&gt;-&lt;ordinal&gt;</code> where <em>ordinal</em> starts from 0 up to replicas-1.</p>

<p>The name of your pods will be:</p>

<pre><code>sharded-svc-0
sharded-svc-1
sharded-svc-2
</code></pre>

<p>and those pods can be reached with a dns-name:</p>

<pre><code>sharded-svc-0.sharded-svc.your-namespace.svc.cluster.local
sharded-svc-1.sharded-svc.your-namespace.svc.cluster.local
sharded-svc-2.sharded-svc.your-namespace.svc.cluster.local
</code></pre>

<p>given that your <em>Headless Service</em> is named <code>sharded-svc</code> and you deploy it in namespace <code>your-namespace</code>.</p>

<h1>Sharding or Partitioning</h1>

<blockquote>
  <p>given multiple replicas of the target service of a frontend -> backend call, I can reliably route traffic for a certain account to a certain endpoint instance.</p>
</blockquote>

<p>What you describe here is that your stateful service is what is called <em>sharded</em> or <em>partitioned</em>. This does not come out of the box from Kubernetes, but you have all the needed <em>building blocks</em> for this kind of service. <em>It may happen that it exists an 3rd party service providing this feature that you can deploy, or it can be developed.</em></p>

<h2>Sharding Proxy</h2>

<p>You can create a service <code>sharding-proxy</code> consisting of one of more pods (possibly from <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/" rel="noreferrer">Deployment</a> since it can be stateless). This app need to watch the pods/service/<a href="https://stackoverflow.com/questions/52857825/what-is-an-endpoint-in-kubernetes">endpoints</a> in your <code>sharded-svc</code> to know where it can route traffic. This can be developed using <a href="https://github.com/kubernetes/client-go" rel="noreferrer">client-go</a> or other alternatives.</p>

<p>This service implements the logic you want in your sharding, e.g. <em>account-nr</em> modulus 3 is routed to the corresponding pod <em>ordinal</em></p>

<p><strong>Update:</strong> There are 3rd party proxies with <strong>sharding</strong> functionallity, e.g. <a href="https://github.com/gojek/weaver" rel="noreferrer">Weaver Proxy</a></p>

<blockquote>
  <p>Sharding request based on headers/path/body fields</p>
</blockquote>

<p>Recommended reading: <a href="https://medium.com/@rbshetty/weaver-proxying-at-scale-b3b8b425a58e" rel="noreferrer">Weaver: Sharding with simplicity</a></p>

<h1>Consuming sharded service</h1>

<p>To consume your sharded service, the clients send request to your <code>sharding-proxy</code> that then apply your <em>routing</em> or <em>sharding logic</em> (e.g. request with <em>account-nr</em> modulus 3 is routed to the corresponding pod <em>ordinal</em>) and forward the request to <em>the replica</em> of <code>sharded-svc</code> that match your logic.</p>

<h1>Alternative Solutions</h1>

<p><strong>Directory Service:</strong> It is probably easier to implement <code>sharded-proxy</code> as a <em>directory service</em> but it depends on your requirements. The clients can ask your <em>directory service</em> to what statefulSet replica should I send <em>account-nr X</em> and your serice reply with e.g. <code>sharded-svc-2</code></p>

<p><strong>Routing logic in client:</strong> The probably most easy solution is to have your <em>routing logic</em>  in the client, and let this logic calculate to what statefulSet replica to send the request.</p>
