<p>I am porting an application to run within k8s.  I have run into an issue with ingress.  I am trying to find a way to limit the number of REST API requests in flight at any given time to each backend pod managed by a deployment.</p>
<p>See the image below the shows the architecture.</p>
<p>Ingress is being managed by nginx-ingress.  For a given set of URL paths, the ingress forwards the request to a service that targets a deployment of REST API backend processes.  The deployment is also managed by an HPA based upon CPU load.</p>
<p>What I want to do is find a way to queue up ingress requests such that there are never more than X requests in flight to any pod running our API backend process.  (ex. only allow 50 requests in flight at once per pod)</p>
<p>Does anyone know how to put a request limit in place like this?</p>
<p>As a bonus question, the next thing I would need to do is have the HPA monitor the request queuing and automatically scale up/down the deployment to match the number of pods to the number of requests currently being processed / queued.  For example if each pod can handle 100 requests in flight at once and we currently have load levels of 1000 requests to handle, then autoscale to 10 pods.</p>
<p>If it is useful, I am also planning to have linkerd in place for this cluster.  Perhaps it has a capability that could help.</p>
<p><a href="https://i.stack.imgur.com/Of3ix.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Of3ix.png" alt="enter image description here" /></a></p>

<p>Autoscaling in Network request requires the custom metrics. Given that you are using the NGINX ingress controller, you can first install prometheus and prometheus adaptor to export the metrics from NGINX ingress controller. By default, NGINX ingress controller has already exposed the prometheus endpoint.</p>
<p>The relation graph will be like this.</p>
<pre><code>NGINX ingress &lt;- Prometheus &lt;- Prometheus Adaptor &lt;- custom metrics api service &lt;- HPA controller
</code></pre>
<p>The arrow means the calling in API. So, in total, you will have three more extract components in your cluster.</p>
<p>Once you have set up the custom metric server, you can scale your app based on the metrics from NGINX ingress. The HPA will look like this.</p>
<pre><code>apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: srv-deployment-custom-hpa
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: srv-deployment
  minReplicas: 1
  maxReplicas: 100
  metrics:
  - type: Pods
    pods:
      metricName: nginx_srv_server_requests_per_second
      targetAverageValue: 100
</code></pre>
<p>I won't go through the actual implementation here because it will include a lot of environment specific configuration.</p>
<p>Once you have set that up, you can see the HPA object will show up the metrics which is pulling from the adaptor.</p>
<p>For the rate limiting in the <code>Service</code> object level, you will need a powerful service mesh to do so. Linkerd2 is designed to be lightweight so it does not ship with the function in rate limiting. You can refer to this issue under <a href="https://github.com/linkerd/linkerd2/issues/4649" rel="nofollow noreferrer">linkerd2</a>. The maintainer rejected to implement the rate limiting in the service level. They would suggest you to do this on <code>Ingress</code> level instead.</p>
<p>AFAIK, Istio and some advanced serivce mesh provides the rate limiting function. In case you haven't deployed the linkerd as your service mesh option, you may try Istio instead.</p>
<p>For Istio, you can refer <a href="https://istio.io/v1.4/docs/tasks/policy-enforcement/rate-limiting/" rel="nofollow noreferrer">this document</a> to see how to do the rate limiting. But I need to let you know that Istio with NGINX ingress may cause you a trouble. Istio is shipped with its own ingress controller. You will need to have extra work for making it work.</p>
<p>To conclude, if you can use the HPA with custom metrics in the number of requests, it will be the quick solution to resolve your issue in traffic control. Unless you still have a really hard time with the traffic control, you will then need to consider the <code>Service</code> level rate limiting.</p>
