<p>I have one master service and multiple slave services. The master service continuously polls a topic using subscriber from Google PubSub. The Slave services are REST APIs. Once the master service receives a message, it delegates the message to a slave service. Currently I'm using ClusterIP service in Kubernetes. Some of my requests are long running and some are pretty short.</p>

<p>I happen to observe that sometimes if there's a short running request while a long running request is in process, it has to wait until the long running request to finish even though many pods are available without serving any traffic. I think it's due to the round robin load balancing. I have been trying to find a solution and looked into approaches like setting up external HTTP load balancer with ingress and internal HTTP load balancer. But I'm really confused about the difference between these two and which one applies for my use case. Can you suggest which of the approaches would solve my use case?</p>

backprop7
<h3>TL;DR</h3>
<p>assuming you want 20% of the traffic to go to <code>x</code> service and the rest 80% to <code>y</code> service. create 2 ingress files for each of the 2 targets, <strong>with same host name</strong>, the only difference is that one of them will carry the following <strong>ingress annotations</strong>: <a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary" rel="nofollow noreferrer">docs</a></p>
<pre><code>nginx.ingress.kubernetes.io/canary: &quot;true&quot; #--&gt; tell the controller to not create a new vhost

nginx.ingress.kubernetes.io/canary-weight: &quot;20&quot; #--&gt; route here 20% of the traffic from the existing vhost 
</code></pre>
<hr />
<h3>WHY &amp; HOW TO</h3>
<p>weighted routing is a bit beyond the <code>ClusterIP</code>. as you said yourself, its time for a new player to enter the game - an <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress" rel="nofollow noreferrer">ingress controller</a>.</p>
<p>this is a k8s abstraction for a <a href="https://www.nginx.com/resources/glossary/load-balancing/" rel="nofollow noreferrer"><code>load balancer</code></a> - a powerful server sitting in front of your app and routing the traffic between the <code>ClusterIP</code>s.</p>
<p><a href="https://cloud.google.com/community/tutorials/nginx-ingress-gke" rel="nofollow noreferrer">install ingress controller on gcp cluster</a></p>
<p>once you have it installed and running, use its <strong>canary</strong> feature to perform a weighted routing. this is done using the following annotations:</p>
<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: http-svc
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: &quot;true&quot;
    nginx.ingress.kubernetes.io/canary-weight: &quot;20&quot;
spec:
  rules:
  - host: echo.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
</code></pre>
<p><a href="https://www.elvinefendi.com/2018/11/25/canary-deployment-with-ingress-nginx.html" rel="nofollow noreferrer">here</a> is the full guide.</p>
<h3>External vs internal load balancing</h3>
<p>(this is the relevant definition from google cloud docs but the concept is similar among other cloud providers)</p>
<blockquote>
<p>GCP's load balancers can be divided into external and internal load
balancers. External load balancers distribute traffic coming from the
internet to your GCP network. Internal load balancers distribute
traffic within your GCP network.</p>
</blockquote>
<p><a href="https://cloud.google.com/load-balancing/docs/load-balancing-overview" rel="nofollow noreferrer">https://cloud.google.com/load-balancing/docs/load-balancing-overview</a></p>
