<p>My Environment: Mac dev machine with latest Minikube/Docker</p>

<p>I built (locally) a simple docker image with a simple Django REST API "hello world".I'm running a deployment with 3 replicas. This is my <code>yaml</code> file for defining it:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: myproj-app-service
  labels:
    app: myproj-be
spec:
  type: LoadBalancer
  ports:
    - port: 8000
  selector:
    app: myproj-be
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myproj-app-deployment
  labels:
    app: myproj-be
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myproj-be
  template:
    metadata:
      labels:
        app: myproj-be
    spec:
      containers:
        - name: myproj-app-server
          image: myproj-app-server:4
          ports:
            - containerPort: 8000
          env:
            - name: DATABASE_URL
              value: postgres://myname:@10.0.2.2:5432/myproj2
            - name: REDIS_URL
              value: redis://10.0.2.2:6379/1
</code></pre>

<p>When I apply this <code>yaml</code> it generates things correctly.
  - one deployment
  - one service
  - three pods</p>

<p>Deployments:</p>

<pre><code>NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
myproj-app-deployment   3/3     3            3           79m
</code></pre>

<p>Services:</p>

<pre><code>NAME               TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
kubernetes         ClusterIP      10.96.0.1     &lt;none&gt;        443/TCP          83m
myproj-app-service   LoadBalancer   10.96.91.44   &lt;pending&gt;     8000:31559/TCP   79m
</code></pre>

<p>Pods:</p>

<pre><code>NAME                                   READY   STATUS    RESTARTS   AGE
myproj-app-deployment-77664b5557-97wkx   1/1     Running   0          48m
myproj-app-deployment-77664b5557-ks7kf   1/1     Running   0          49m
myproj-app-deployment-77664b5557-v9889   1/1     Running   0          49m
</code></pre>

<p>The interesting thing is that when I <code>SSH</code> into the <code>Minikube</code>, and hit the service using <code>curl 10.96.91.44:8000</code> it respects the <code>LoadBalancer</code> type of the service and rotates between all three pods as I hit the endpoints time and again. I can see that in the returned results which I have made sure to include the HOSTNAME of the pod.</p>

<p>However, when I try to access the service from my Hosting Mac -- using <code>kubectl port-forward service/myproj-app-service 8000:8000</code> -- Every time I hit the endpoint, I get the same pod to respond. It doesn't load balance. I can see that clearly when I <code>kubectl logs -f &lt;pod&gt;</code> to all three pods and only one of them is handling the hits, as the other two are idle... </p>

<p>Is this a <code>kubectl port-forward</code> limitation or issue? or am I missing something greater here?</p>

<p><code>kubectl port-forward</code> looks up the first Pod from the Service information provided on the command line and forwards directly to a Pod rather than forwarding to the ClusterIP/Service port. The cluster doesn't get a chance to load balance the service like regular service traffic.</p>
<p>The kubernetes API only provides <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#create-connect-portforward-pod-v1-core" rel="noreferrer">Pod port forward operations</a> (<code>CREATE</code> and <code>GET</code>). Similar API operations don't exist for Service endpoints.</p>
<h3><code>kubectl</code> code</h3>
<p>Here's a little bit of the flow from the <code>kubectl</code> code that seems to back that up (I'll just add that Go isn't my primary language)</p>
<p>The <a href="https://github.com/kubernetes/kubectl/blob/c53c16a548eb34f54f673efee2b9b09c52ec15b5/pkg/cmd/portforward/portforward.go#L225" rel="noreferrer">portforward.go <code>Complete</code> function</a> is where <code>kubectl portforward</code> does the first look up for a pod from options via <a href="https://github.com/kubernetes/kubectl/blob/c53c16a548eb34f54f673efee2b9b09c52ec15b5/pkg/cmd/portforward/portforward.go#L254" rel="noreferrer"><code>AttachablePodForObjectFn</code></a>:</p>
<p>The <code>AttachablePodForObjectFn</code> is defined as <code>attachablePodForObject</code> in <a href="https://github.com/kubernetes/kubectl/blob/6d12ae1ac20bee2d3b5fb7a664de76d7fc134a63/pkg/polymorphichelpers/interface.go#L39-L40" rel="noreferrer">this interface</a>, then here is the <a href="https://github.com/kubernetes/kubectl/blob/6d12ae1ac20bee2d3b5fb7a664de76d7fc134a63/pkg/polymorphichelpers/attachablepodforobject.go" rel="noreferrer"><code>attachablePodForObject</code> function</a>.</p>
<p>To my (inexperienced) Go eyes, it appears the <a href="https://github.com/kubernetes/kubectl/blob/6d12ae1ac20bee2d3b5fb7a664de76d7fc134a63/pkg/polymorphichelpers/attachablepodforobject.go" rel="noreferrer"><code>attachablePodForObject</code></a> is the thing <code>kubectl</code> uses to look up a Pod to from a Service defined on the command line.</p>
<p>Then from there on everything deals with filling in the Pod specific <a href="https://github.com/kubernetes/kubectl/blob/c53c16a548eb34f54f673efee2b9b09c52ec15b5/pkg/cmd/portforward/portforward.go#L46-L58" rel="noreferrer"><code>PortForwardOptions</code></a> (which doesn't include a service) and is passed to the kubernetes API.</p>
