<p>I have been getting more and more into k8s and I am trying some stuff on a personal VPS I have.
I have created a Deployment of a POD that uses another Service internally.
I would love to validate that these two services are being somewhat loadbalanced.</p>
<p>Here was my attempt to create this:
I have a simple service which I called <strong>metric-test</strong> which only has one endpoint that counts how many times it was called, logs it and returns this information.
For this I used the microframework <strong>jooby</strong> since I had familiarity with it and could get a fast start.</p>
<p>The code of this simple app can be <a href="https://github.com/Flavsditz/k8s-balance-test" rel="nofollow noreferrer">found on github</a></p>
<p>Also on the repository I have added the <strong>deployment.yaml</strong> file with which I use to push it to my local version of <strong>minikube</strong> (which simulates my k8s env).</p>
<p>Steps taken:</p>
<ol>
<li>Use this cmd to compile docker image into minikube repo: <code>eval $(minikube docker-env)</code></li>
<li>I now build the project's docker image with <code>docker build . -t metric-test1</code></li>
<li>I then apply the deployment file with <code>kubectl apply -f deployment.yaml</code> (the file is also on the github link</li>
</ol>
<p>This gives me a Service of type <strong>ClusterIP</strong> (which is what I want since it should not be accessible from outside) and 2 PODS containing the jooby code. Here is the <code>deployment.yaml</code> file:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: metric-test
  labels:
    run: metric-test
spec:
  ports:
    - port: 3000
      protocol: TCP
  selector:
    run: metric-test

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metric-test
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      run: metric-test
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        run: metric-test
    spec:
      containers:
        - image: metric-test1
          imagePullPolicy: Never
          name: data-api
          ports:
            - containerPort: 3000
              protocol: TCP
              name: 3000tcp
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
</code></pre>
<p>Alright everything working! I just set up a port-forwarding so I can access the service:
<code>kubectl port-forward service/metric-test 3000:3000</code></p>
<p>And use this script to fire a lot of requests at the service:</p>
<pre class="lang-sh prettyprint-override"><code>#!/bin/bash
target=${1:-http://localhost:3000}
while true # loop forever, until ctrl+c pressed.
do
        for i in $(seq 100) # perfrom the inner command 100 times.
        do
                curl $target &gt; /dev/null &amp; # send out a curl request, the &amp; indicates not to wait for the response.
        done

        wait # after 100 requests are sent out, wait for their processes to finish before the next iteration.
done
</code></pre>
<p>I am now seeing <strong>ALL</strong> of the requests being handled by only one of the pods while the other is just sitting there idle.</p>
<p>I went through the documentation (<a href="https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types" rel="nofollow noreferrer">here</a>) but to be honest I came out actually with more questions than answers. That is why I tried to create a simplified scenario to test these things out.</p>
<p><strong>Questions</strong></p>
<ol>
<li>Can anyone help me out?</li>
<li>What am I missing? How can I achieve this load-balancing without exposing the service through the internet. I just want it to be available for other pods in the cluster.</li>
<li>(Bonus) The front-end services that are being served by an Ingress should all be properly load-balanced (right?)</li>
</ol>
<p>Note: As far as I understood using <code>LoadBalancer</code> and <code>Ingress</code> you can actually achieve load-balancing, however, it also exposes to the outside.</p>
<h2><strong>EDIT 1</strong></h2>
<p>More info on the deployment:
Result of <code>kubectl get po</code></p>
<pre><code>NAME                          READY   STATUS    RESTARTS   AGE
metric-test-f89bfbf86-ccrj8   1/1     Running   0          16h
metric-test-f89bfbf86-kl7qg   1/1     Running   0          16h
</code></pre>
<p>Here is a picture of the logs of both after running the curl script for a bit:
<a href="https://i.stack.imgur.com/fRoUv.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fRoUv.png" alt="Logs of both PODS" /></a></p>
<h2><strong>EDIT FOR SOLUTION</strong></h2>
<p>As m303945 said, the load balancing doesn't work out when we are using the port-forwarding.</p>
<p>In order to validate this, and any future tests I might want to do I did the following:</p>
<p>I've ran the following command in my terminal:</p>
<pre><code>kubectl run -it --rm --restart=Never --image=alpine handytools -n ${1:-default} -- /bin/ash
</code></pre>
<p>which creates an alpine-based container and gives me shell access. At that point however, I cannot use curl since it is not installed. So for that I ran:</p>
<ul>
<li><code>apk update</code></li>
<li><code>apk add curl</code></li>
</ul>
<p>once I had that I've modified my previous bash script from above to run on this pod and try to hit the service I set up:</p>
<pre class="lang-sh prettyprint-override"><code>#!/bin/ash
target=${1:-http://metric-test:3000}
for i in $(seq 5) # loop 5 times to generate 500 calls.
do
        for i in $(seq 100) # perfrom the inner command 100 times.
        do
                curl $target &gt; /dev/null &amp; # send out a curl request, the &amp; indicates not to wait for the response.
        done

        wait # after 100 requests are sent out, wait for their processes to finish before the next iteration.
done
</code></pre>
<p>The modifications included pointing to the service instead of the &quot;localhost&quot; and no port is needed. Also the alpine uses the <code>ash</code> and not the <code>bash</code> shell. I also ran 500 requests instead of infinity.</p>
<p>As you can see, running the above (which you can create using <code>vi</code> in alpine) I get a nice even load distribution!!!
<a href="https://i.stack.imgur.com/oCBst.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/oCBst.png" alt="even distribution of calls" /></a></p>
<p>Thanks again for user m303945 for pointing me in the right direction</p>

fditz
<p>If I remember correctly, TCP load balancing was not working when using port forwarding. Try to run the script from a container inside k8s instead of doing port forwarding.</p>
