<p>I have 2 different microservices deployed as backend in minikube, call it deployment A and deployment B. Both these deployments have a different replica of pods running.</p>
<p>Deployment B is exposed as service B. Pods of deployment A call deployment B pods via service B which is of ClusterIP type.</p>
<p>The pods of deployment B have a scrapyd application running inside them with scraping spiders deployed in them. Each celery worker( pods of deployment A) takes a task from a redis queue and calls scrapyd service to schedule spiders on them.</p>
<p>Everything works fine but after I scale the application (deployment A and B seperately),  I observe that the resource consumption is not uniform, using <code>kubectl top pods</code>
I observe that some of the pods of deployment B are not used at all. The pattern I observe is that only those pods of deployment B, that are up and running after all the pods of deployment A are up, are never utilized.</p>
<p>Is it normal behavior? I suspect the connection between pods of deployment A and B is persistent I am confused as to why request handling by pods of deployment B is not uniformly distributed? Sorry for the naive question. I am new to this field.</p>
<p>The manifest for <strong>deployment A</strong> is :</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: celery-worker
  labels:
    deployment: celery-worker
spec:
  replicas: 1
  selector:
    matchLabels:
      pod: celery-worker
  template:
    metadata:
      labels:
        pod: celery-worker
    spec:
      containers:
        - name: celery-worker
          image: celery:latest
          imagePullPolicy: Never
          command: ['celery', '-A', 'mysite', 'worker', '-E', '-l', 'info',]
          resources:
            limits:
              cpu: 500m
            requests:
              cpu: 200m
      terminationGracePeriodSeconds: 200
</code></pre>
<p>and that of <strong>deployment B</strong> is</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: scrapyd
  labels:
    app: scrapyd
spec:
  replicas: 1
  selector:
    matchLabels:
      pod: scrapyd
  template:
    metadata:
      labels:
        pod: scrapyd
    spec:
      containers:
        - name: scrapyd
          image: scrapyd:latest
          imagePullPolicy: Never
          ports:
            - containerPort: 6800
          resources:
            limits:
              cpu: 800m
            requests:
              cpu: 800m
      terminationGracePeriodSeconds: 100

---
kind: Service
apiVersion: v1
metadata:
  name: scrapyd
spec:
  selector:
    pod: scrapyd
  ports:
  - protocol: TCP
    port: 6800
    targetPort: 6800

</code></pre>
<p>Output of <code>kubectl top pods</code> :</p>
<p><a href="https://i.stack.imgur.com/PDCOI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PDCOI.png" alt="enter image description here" /></a></p>

<p>So the solution to the above problem that I figured out is as follows :</p>
<p>In the current setup, install linkerd using this link. <a href="https://linkerd.io/2/getting-started/" rel="nofollow noreferrer">Linkerd Installation in Kubernetes</a></p>
<p>After that inject the linkerd proxy into celery deployment as follows :</p>
<pre><code>cat celery/deployment.yaml | linkerd inject - | kubectl apply -f -
</code></pre>
<p>This ensures that the requests from celery are first passed on to this proxy and then Load balanced directly to scrapy at L7 Layer. In this case, kube-proxy is by passed and the default load balancing over L4 is no longer functional.</p>
