<p>I am using Kubernetes in Azure with Virtual Nodes this is a plugin that creates virtual nodes using Azure Container Instances.</p>
<p>The instructions to set this up require creating a AKSNet/AKSSubnet which seems to automatically come along with A VMSS called something like. aks-control-xxx-vmss I followed the instruction on the link below.</p>
<p><a href="https://learn.microsoft.com/en-us/azure/aks/virtual-nodes-cli" rel="nofollow noreferrer">https://learn.microsoft.com/en-us/azure/aks/virtual-nodes-cli</a></p>
<p>This comes with a single instance VM that I am being charged full price for regardless of container instances I create and I am charged extra for every container instance I provision onto my virtual node pool even if they should fit on just 1 VM. These resources do not seem to be related.</p>
<p>I am currently having this out as unexpected billing with Microsoft but the process has been very slow so I am reverting to here to find out if anyone else has had this experience?</p>
<p>The main questions I have are:</p>
<ul>
<li>Can I use Azure Container Instances without the VMSS?</li>
<li>If not can I somehow make this VM visible to my cluster so I can at least use it to
provision containers onto and get some value out of it?</li>
<li>Have I just done something wrong?</li>
</ul>
<p>Update, NB: this is not my control node that is a B2s which I can see my system containers running on.</p>
<p>Any advice would be a great help.</p>

Lenny D
<blockquote>
<p>Can I use Azure Container Instances without the VMSS?</p>
</blockquote>
<p>In an AKS cluster currently you <em><strong>cannot</strong></em> have virtual nodes without a <strong>node pool</strong> of type <code>VirtualMachineScaleSets</code> or <code>AvailabilitySet</code>. An AKS cluster has at least one node, an Azure virtual machine (VM) that runs the Kubernetes node components and container runtime. [<a href="https://learn.microsoft.com/en-us/azure/aks/concepts-clusters-workloads#nodes-and-node-pools" rel="nofollow noreferrer">Reference</a>] Every AKS cluster must contain at least one system node pool with at least one node. System node pools serve the primary purpose of hosting critical system pods such as <code>CoreDNS</code>, <code>kube-proxy</code> and <code>metrics-server</code>. However, application pods can be scheduled on system node pools if you wish to only have one pool in your AKS cluster.</p>
<p>For more information on System Node Pools please check <a href="https://learn.microsoft.com/en-us/azure/aks/use-system-pools" rel="nofollow noreferrer">this document</a>.</p>
<p>In fact, if you run <code>kubectl get pods -n kube-system -o wide</code> you will see all the system pods running on the VMSS-backed node pool node including the aci-connector-linux-xxxxxxxx-xxxxx pod which connects the cluster to the virtual node, as shown below:</p>
<pre><code>NAME                                  READY   STATUS    RESTARTS   AGE   IP            NODE                                NOMINATED NODE   READINESS GATES
aci-connector-linux-859d9ff5-24tgq    1/1     Running   0          49m   10.240.0.30   aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
azure-cni-networkmonitor-7zcvf        1/1     Running   0          58m   10.240.0.4    aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
azure-ip-masq-agent-tdhnx             1/1     Running   0          58m   10.240.0.4    aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
coredns-autoscaler-6699988865-k7cs5   1/1     Running   0          58m   10.240.0.31   aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
coredns-d4866bcb7-4r9tj               1/1     Running   0          49m   10.240.0.12   aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
coredns-d4866bcb7-5vkhc               1/1     Running   0          58m   10.240.0.28   aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
coredns-d4866bcb7-b7bzg               1/1     Running   0          49m   10.240.0.11   aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
coredns-d4866bcb7-fltbf               1/1     Running   0          49m   10.240.0.29   aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
coredns-d4866bcb7-n94tg               1/1     Running   0          57m   10.240.0.34   aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
konnectivity-agent-7564955db-f4fm6    1/1     Running   0          58m   10.240.0.4    aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
kube-proxy-lntqs                      1/1     Running   0          58m   10.240.0.4    aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
metrics-server-97958786-bmmv9         1/1     Running   1          58m   10.240.0.24   aks-nodepool1-29819654-vmss000000   &lt;none&gt;           &lt;none&gt;
</code></pre>
<p>However, you can deploy <a href="https://learn.microsoft.com/en-us/azure/container-instances/container-instances-overview" rel="nofollow noreferrer">Azure Container Instances</a> [<a href="https://learn.microsoft.com/en-us/azure/container-instances/container-instances-quickstart" rel="nofollow noreferrer">How-to</a>] without an AKS cluster altogether. For scenarios where you need full container orchestration, including service discovery across multiple containers, automatic scaling, and coordinated application upgrades, we recommend <a href="https://learn.microsoft.com/en-us/azure/aks/" rel="nofollow noreferrer">Azure Kubernetes Service (AKS)</a>.</p>
<blockquote>
<p>If not can I somehow make this VM visible to my cluster so I can at least use it to provision containers onto and get some value out of it?</p>
</blockquote>
<p>Absolutely, you can. In fact if you do a <code>kubectl get nodes</code> and the node from the VMSS-backed node pool (in your case aks-control-xxx-vmss-x) shows <code>STATUS</code> as Ready, then it is available to the <code>kube-scheduler</code> for <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/" rel="nofollow noreferrer">scheduling</a> workloads. Please check <a href="https://kubernetes.io/docs/concepts/architecture/nodes/#condition" rel="nofollow noreferrer">this document</a>.</p>
<p>If you do a <code>kubectl describe node virtual-node-aci-linux</code> you should find the following in the output:</p>
<pre><code>...
Labels:             alpha.service-controller.kubernetes.io/exclude-balancer=true
                    beta.kubernetes.io/os=linux
                    kubernetes.azure.com/managed=false
                    kubernetes.azure.com/role=agent
                    kubernetes.io/hostname=virtual-node-aci-linux
                    kubernetes.io/role=agent
                    node-role.kubernetes.io/agent=
                    node.kubernetes.io/exclude-from-external-load-balancers=true
                    type=virtual-kubelet
...
Taints:             virtual-kubelet.io/provider=azure:NoSchedule
...
</code></pre>
<p>In <a href="https://learn.microsoft.com/en-us/azure/aks/virtual-nodes-cli" rel="nofollow noreferrer">the document</a> that you are following, in the <a href="https://learn.microsoft.com/en-us/azure/aks/virtual-nodes-cli#deploy-a-sample-app" rel="nofollow noreferrer">Deploy a sample app section</a> to schedule the container on the virtual node, a <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/" rel="nofollow noreferrer">nodeSelector</a> and <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="nofollow noreferrer">toleration</a> are defined in the <a href="https://learn.microsoft.com/en-us/azure/aks/virtual-nodes-cli#deploy-a-sample-app" rel="nofollow noreferrer">Deploy a sample app section</a> as follows:</p>
<pre><code>...
nodeSelector:
        kubernetes.io/role: agent
        beta.kubernetes.io/os: linux
        type: virtual-kubelet
      tolerations:
      - key: virtual-kubelet.io/provider
        operator: Exists
      - key: azure.com/aci
        effect: NoSchedule
</code></pre>
<p>If you remove this part from the <em>Deployment</em> manifest, or do not specify this part in the manifest of a workload that you are deploying, then the corresponding resource(s) will be scheduled on a VMSS-backed node.</p>
<blockquote>
<p>Have I just done something wrong?</p>
</blockquote>
<p>Maybe you can evaluate the answer to this based on my responses to your earlier questions. However, here's a little more to help you understand:</p>
<p>If a node doesn't have sufficient compute resources to run a requested pod, that pod can't progress through the scheduling process. The pod can't start unless additional compute resources are available within the node pool.</p>
<p>When the cluster autoscaler notices pods that can't be scheduled because of node pool resource constraints, the number of nodes within the node pool is increased to provide the additional compute resources. When those additional nodes are successfully deployed and available for use within the node pool, the pods are then scheduled to run on them.</p>
<p>If your application needs to scale rapidly, some pods may remain in a state waiting to be scheduled until the additional nodes deployed by the cluster autoscaler can accept the scheduled pods. For applications that have high burst demands, you can scale with virtual nodes and Azure Container Instances.</p>
<p>This however <strong>does not mean that we can dispense with the VMSS or Availability Set backed node pools</strong>.</p>
