<p>We are developing a simulation software which is deployed and scaled between multiple pods using kubernetes.
When a user makes a simulation request, a pod is selected which starts doing the job and is considered as busy.
When another user makes a simulation request, it should be routed to the next free pod.
Currently, a busy pod is often selected (even though there are free ones) as kubernetes does not know which pods are busy/free.</p>
<p>Is it possible to balance requests in such way that a free pod is always selected?
(Assuming that each app instance inside a pod exposes an HTTP endpoint which tells it's current busy/free status)</p>

Lukas Morkūnas
<p>Another solution to the one suggested in above answer is to make use of Headless Service and some reverse-proxy/load balancer like HAProxy(I used it).</p>
<p>So you can have a headless service or change the existing service which is  exposing your pods like below. It is the configuration <strong>clusterIP:None</strong> which will be responsible for creating a headless service.</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: some-service
  namespace: dev
  labels:
    app: &lt;labels here&gt;
spec:
  ports:
    - port: &lt;port&gt;
      targetPort: &lt;tport&gt;
  selector:
    app: &lt;selectors here&gt;
  clusterIP: None
</code></pre>
<p>Then you can deploy HAProxy like below</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: haproxy-headless
  namespace: dev
  labels:
    app: haproxy-headless
spec:
  replicas: 1
  selector:
    matchLabels:
      app: haproxy-headless
  template:
    metadata:
      labels:
        app: haproxy-headless
    spec:
      containers:
      - name: haproxy-headless
        image: haproxy:1.9
        ports:
        - containerPort: 8888
          name: management
        - containerPort: 8085
          name: http
        volumeMounts:
         - name: haproxy-config
           mountPath: &quot;/usr/local/etc/haproxy/haproxy.cfg&quot;
      volumes:
        - name: haproxy-config
          configMap:
            name: haproxy-config
</code></pre>
<p>ConfigMap  for Haproxy:</p>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: haproxy-config
  namespace: dev
data:
   haproxyconfig.cfg: |
     
     defaults
       mode tcp
       log global
       option httplog
       retries 5
       timeout connect 10s
       timeout client 300s
       timeout server 300s 
     
     resolvers test_resolver
       nameserver dns1 &lt;your dns server address&gt;
       resolve_retries 30
       timeout retry 2s
       hold valid 10s
       accepted_payload_size 8192
     
     frontend stats
        mode http
        bind :8888
        stats enable
        stats uri /
        stats refresh 15s
     
     frontend test_fe
       bind :8085
       timeout client 60s
       default_backend test_be
       
     backend test_be
       balance leastconn
       server-template srv 7 &lt;service-name&gt;.&lt;your namespace&gt;.svc.cluster.local:6311 check resolvers test_resolver
</code></pre>
<p>Important thing here to understand how we configure our HAProxy here server-template is being used for service discovery using the DNS of K8s and you can use <strong>balance leastconn</strong> to make sure the least used server is used.</p>
<p>In the end a service would be required for the HAProxy</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: haproxy-service
  namespace: dev
spec:
  selector:
    app: haproxy-headless
  ports:
    - name: be
      protocol: TCP
      port: 6311
      targetPort: 8085
    - name: mngmnt
      protocol: TCP
      port: 8888
      targetPort: 8888
  type: &lt;Type as per requirement&gt;
</code></pre>
<p>Now use the above service whenever you want to access the app. Maybe you can read about how to use HAProxy in k8s for load balancing it basically doing service discovery + load balancing.</p>
<p>Maybe a few configurations shown in the yaml are not correct or mismatched but i hope concept is clear about how to achieve this.</p>
