<p>In Azure Kubernetes Service, the number of Azure Disks you can attach to a Node depends on the VM size. Eg. <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/dv4-dsv4-series" rel="nofollow noreferrer">https://learn.microsoft.com/en-us/azure/virtual-machines/dv4-dsv4-series</a>. See the data disks column.</p>
<p>I have pods that attach a disk using the <code>disk.csi.azure.com</code> provisioner.
This works as long as there are less then 4 concurrent pods like that.
The fifth stalls and errors out, because the VM size we use only supports 4 disks.</p>
<p>How can I schedule pods with the disk limit in mind, such that the auto scaler scales up the cluster when the disk limit is reached on all the nodes?</p>
<p>Based on <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#extended-resources" rel="nofollow noreferrer">this</a> I hoped AKS would expose this limit as an extended resource, but there is no hint of it in the <code>capacity</code> status of the nodes.
<a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity" rel="nofollow noreferrer">Affinities</a> seem to be binary rules for putting pods on the node; I see no way to model &quot;at most x pods that do y can go to a node&quot;.</p>
<p>Edit for context: The pods are CI jobs generated by gitlab-runner. I want to use persistent volumes as build areas to avoid having to clone the entire project from scratch in every job. From the point of view of Kubenernetes this looks like dynamically generated Pods, each of which requests to mount one of a previously defined set of Persistent Volume Claims. The PVC:s are fungible (they cache the same clone), but the configuration forces us to request one by some specific name for each pod.</p>
<p>Update: I am exploring a solution based on <a href="https://kubernetes.io/docs/tasks/administer-cluster/extended-resource-node/" rel="nofollow noreferrer">Extended Resources</a>. I tried setting them based on the VM size label via Kyverno, but it seems that Kyverno's support for mutating existing resources does not work for the status subresource. Or at least my configuration successfully updates labels on existing nodes, and extended resources on new nodes, but does not set extended resources on existing nodes. Here's my attempt:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: azure-disk-resource-limit
spec:
  mutateExistingOnPolicyUpdate: true
  background: true
  rules:
  - name: add-disk-limit
    match:
      any:
      - resources:
          kinds:
          - Node
          - Node/status
    mutate:
      targets:
        - apiVersion: v1
          kind: Node
        - apiVersion: v1
          kind: Node/status
      patchStrategicMerge:
        metadata:
          labels:
            kyverno-dummy: &quot;babar1&quot;
        status:
          capacity:
            kyverno-dummy: 9
#      patchesJson6902: |-
#          - path: &quot;/status/capacity/kyverno-dummy&quot;
#            op: add
#            value: 8
</code></pre>
<p>NB: I had to edit the Kyverno settings to enable changes to Nodes: 1) I changed the resourceFilter and removed the filter that was hiding Node. 2) I gave the background and admission controllers the right to edit nodes:</p>
<pre class="lang-yaml prettyprint-override"><code># In values.yml for the kyverno chart:
admissionController:
  rbac:
    clusterRole:
      extraResources:
        - apiGroups:
            - ''
          resources:
            - nodes
            - nodes/status
            - namespaces
          verbs:
            - update
            - patch
backgroundController:
  nodeSelector:
    kubernetes.io/os: linux
  rbac:
    clusterRole:
      extraResources:
        - apiGroups:
            - ''
          resources:
            - nodes
            - nodes/status
            - namespaces
          verbs:
            - update
            - patch
</code></pre>
<p>I'm at the point where I'm considering the creation of a custom controller for setting the extended resource, but that doesn't feel sane.</p>
<p>PS. I found a UserVoice requesting AKS would add the limit automatically: <a href="https://feedback.azure.com/d365community/idea/02e94731-f824-ec11-b6e6-000d3a4f0da0" rel="nofollow noreferrer">https://feedback.azure.com/d365community/idea/02e94731-f824-ec11-b6e6-000d3a4f0da0</a>.</p>

Mitten.O
<p>You can think about using S3 buckets instead of attached disks, they may be even better suited for high (unknown) number of pods requiring storage with pod-specific naming conventions.</p>
<p>Another possibility is to try out a topology spread constraint of the <code>topologyKey</code> &quot;node&quot;, e.g.</p>
<pre><code>kind: Pod
apiVersion: v1
metadata:
  name: mypod
  labels:
    foo: bar
spec:
  topologySpreadConstraints:
  - maxSkew: 4
    topologyKey: node
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        foo: bar
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.1
</code></pre>
<p><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#example-one-topologyspreadconstraint" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#example-one-topologyspreadconstraint</a></p>
