<p>It's now quite a few days that I'm trying to configure the cluster on AKS but I keep jumping between parts of the docs, various questions here on SO, articles on Medium.. all to keep failing at it.</p>
<p>The goal is get a static ip with a dns that I can use to connect my apps to the server deployed on AKS.</p>
<p>I have created via terraform the infrastructure which consists of a resource group in which I created a Public IP and the AKS cluster, so far so good.</p>
<p><a href="https://i.stack.imgur.com/eGOrm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/eGOrm.png" alt="resource groups" /></a></p>
<p><a href="https://i.stack.imgur.com/Vp4CH.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Vp4CH.png" alt="fixi-resource-group" /></a></p>
<p><a href="https://i.stack.imgur.com/saHuz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/saHuz.png" alt="MC_fixit-resource-group" /></a></p>
<p>After trying to use the ingress controller that gets installed when you use the option <code>http_application_routing_enabled = true</code> on cluster creation which the docs are discouraging for production <a href="https://learn.microsoft.com/en-us/azure/aks/http-application-routing" rel="nofollow noreferrer">https://learn.microsoft.com/en-us/azure/aks/http-application-routing</a>, I'm trying the recommended way and install the ingress-nginx controller via Helm <a href="https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli" rel="nofollow noreferrer">https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli</a>.</p>
<p>In terraform I'm installing it all like this</p>
<h1>resource group and cluster</h1>
<pre><code>resource &quot;azurerm_resource_group&quot; &quot;resource_group&quot; {
  name     = var.resource_group_name
  location = var.location
    tags = {
    Environment = &quot;Test&quot;
    Team = &quot;DevOps&quot;
  }
}


resource &quot;azurerm_kubernetes_cluster&quot; &quot;server_cluster&quot; {
  name                = &quot;server_cluster&quot;
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name
  dns_prefix          = &quot;fixit&quot;
  kubernetes_version = var.kubernetes_version
  # sku_tier = &quot;Paid&quot;

  default_node_pool {
    name       = &quot;default&quot;
    node_count = 1
    min_count = 1
    max_count = 3
    # vm_size    = &quot;standard_b2s_v5&quot;
    # vm_size    = &quot;standard_e2bs_v5&quot;
    vm_size    = &quot;standard_b4ms&quot;
    type = &quot;VirtualMachineScaleSets&quot;
    enable_auto_scaling = true
    enable_host_encryption = false
    # os_disk_size_gb = 30

    
    # enable_node_public_ip = true

  }

  service_principal {
    client_id = var.sp_client_id
    client_secret = var.sp_client_secret
  }

  tags = {
    Environment = &quot;Production&quot;
  }

  linux_profile {
    admin_username = &quot;azureuser&quot;
    ssh_key {
        key_data = var.ssh_key
    }
  }
  network_profile {
      network_plugin = &quot;kubenet&quot;
      load_balancer_sku = &quot;standard&quot;
      # load_balancer_sku = &quot;basic&quot;
    
  }
  # http_application_routing_enabled = true
  http_application_routing_enabled = false

  
}

</code></pre>
<h1>public ip</h1>
<pre><code>resource &quot;azurerm_public_ip&quot; &quot;public-ip&quot; {
  name                = &quot;fixit-public-ip&quot;
  location            = var.location
  resource_group_name = var.resource_group_name
  allocation_method   = &quot;Static&quot;
  domain_name_label = &quot;fixit&quot;
  sku = &quot;Standard&quot;
}
</code></pre>
<h1>load balancer</h1>
<pre><code>resource &quot;kubernetes_service&quot; &quot;cluster-ingress&quot; {
  metadata {
    name = &quot;cluster-ingress-svc&quot;
    annotations = {
      &quot;service.beta.kubernetes.io/azure-load-balancer-resource-group&quot; = &quot;fixit-resource-group&quot;

      # Warning  SyncLoadBalancerFailed  2m38s (x8 over 12m)  service-controller  Error syncing load balancer: 
      # failed to ensure load balancer: findMatchedPIPByLoadBalancerIP: cannot find public IP with IP address 52.157.90.236 
      # in resource group MC_fixit-resource-group_server_cluster_westeurope

      # &quot;service.beta.kubernetes.io/azure-load-balancer-resource-group&quot; = &quot;MC_fixit-resource-group_server_cluster_westeurope&quot;



      # kubernetes.io/ingress.class: addon-http-application-routing
    }
  }
  spec {
    # type = &quot;Ingress&quot;
    type = &quot;LoadBalancer&quot;
    load_balancer_ip = var.public_ip_address
    selector = {
      name = &quot;cluster-ingress-svc&quot;
    }
    port {
      name = &quot;cluster-port&quot;
      protocol = &quot;TCP&quot;
      port = 3000
      target_port = &quot;80&quot;
    }

  }
}

</code></pre>
<h1>ingress controller</h1>
<pre><code>resource &quot;helm_release&quot; &quot;nginx&quot; {
  name = &quot;ingress-nginx&quot;
  repository = &quot;https://kubernetes.github.io/ingress-nginx&quot;
  chart = &quot;ingress-nginx&quot;
  namespace = &quot;default&quot;

  set {
    name  = &quot;rbac.create&quot;
    value = &quot;false&quot;
  }

  set {
    name  = &quot;controller.service.externalTrafficPolicy&quot;
    value = &quot;Local&quot;
  }

  set {
    name  = &quot;controller.service.loadBalancerIP&quot;
    value = var.public_ip_address
  }

  set {
    name  = &quot;controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-internal&quot;
    value = &quot;true&quot;
  }
  
#   --set controller.service.annotations.&quot;service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path&quot;=/healthz

  set {
    name  = &quot;controller.service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-health-probe-request-path&quot;
    value = &quot;/healthz&quot;
  }
} 
</code></pre>
<p>but the installation fails with this message from terraform</p>
<pre><code>Warning: Helm release &quot;ingress-nginx&quot; was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│ 
│   with module.ingress_controller.helm_release.nginx,
│   on modules/ingress_controller/controller.tf line 2, in resource &quot;helm_release&quot; &quot;nginx&quot;:
│    2: resource &quot;helm_release&quot; &quot;nginx&quot; {
│ 
╵
╷
│ Error: timed out waiting for the condition
│ 
│   with module.ingress_controller.helm_release.nginx,
│   on modules/ingress_controller/controller.tf line 2, in resource &quot;helm_release&quot; &quot;nginx&quot;:
│    2: resource &quot;helm_release&quot; &quot;nginx&quot; {
</code></pre>
<p>the controller print out</p>
<pre><code>vincenzocalia@vincenzos-MacBook-Air helm_charts % kubectl describe svc ingress-nginx-controller
Name:                     ingress-nginx-controller
Namespace:                default
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.5.1
                          helm.sh/chart=ingress-nginx-4.4.2
Annotations:              meta.helm.sh/release-name: ingress-nginx
                          meta.helm.sh/release-namespace: default
                          service: map[beta:map[kubernetes:map[io/azure-load-balancer-internal:true]]]
                          service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.0.173.243
IPs:                      10.0.173.243
IP:                       52.157.90.236
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31709/TCP
Endpoints:                
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30045/TCP
Endpoints:                
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     32500
Events:
  Type     Reason                  Age                 From                Message
  ----     ------                  ----                ----                -------
  Normal   EnsuringLoadBalancer    32s (x5 over 108s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  31s (x5 over 107s)  service-controller  Error syncing load balancer: failed to ensure load balancer: findMatchedPIPByLoadBalancerIP: cannot find public IP with IP address 52.157.90.236 in resource group mc_fixit-resource-group_server_cluster_westeurope


vincenzocalia@vincenzos-MacBook-Air helm_charts % az aks show --resource-group fixit-resource-group --name server_cluster --query nodeResourceGroup -o tsv
MC_fixit-resource-group_server_cluster_westeurope
</code></pre>
<p>Why is it looking in the <code>MC_fixit-resource-group_server_cluster_westeurope</code> resource group and not in the <code>fixit-resource-group</code> I created for the Cluster, Public IP and Load Balancer?</p>
<p>If I change the controller load balancer ip to the public ip in <code>MC_fixit-resource-group_server_cluster_westeurope</code> then terraform still outputs the same error, but the controller prints out to be correctly assigned to the ip and load balancer</p>
<pre><code>set {
    name  = &quot;controller.service.loadBalancerIP&quot;
    value = &quot;20.73.192.77&quot; #var.public_ip_address
  }
</code></pre>
<pre><code>vincenzocalia@vincenzos-MacBook-Air helm_charts % kubectl get svc
NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
cluster-ingress-svc                  LoadBalancer   10.0.110.114   52.157.90.236   3000:31863/TCP               104m
ingress-nginx-controller             LoadBalancer   10.0.106.201   20.73.192.77    80:30714/TCP,443:32737/TCP   41m
ingress-nginx-controller-admission   ClusterIP      10.0.23.188    &lt;none&gt;          443/TCP                      41m
kubernetes                           ClusterIP      10.0.0.1       &lt;none&gt;          443/TCP                      122m
vincenzocalia@vincenzos-MacBook-Air helm_charts % kubectl describe svc ingress-nginx-controller
Name:                     ingress-nginx-controller
Namespace:                default
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.5.1
                          helm.sh/chart=ingress-nginx-4.4.2
Annotations:              meta.helm.sh/release-name: ingress-nginx
                          meta.helm.sh/release-namespace: default
                          service: map[beta:map[kubernetes:map[io/azure-load-balancer-internal:true]]]
                          service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.0.106.201
IPs:                      10.0.106.201
IP:                       20.73.192.77
LoadBalancer Ingress:     20.73.192.77
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30714/TCP
Endpoints:                
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  32737/TCP
Endpoints:                
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     32538
Events:
  Type    Reason                Age                From                Message
  ----    ------                ----               ----                -------
  Normal  EnsuringLoadBalancer  39m (x2 over 41m)  service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   39m (x2 over 41m)  service-controller  Ensured load balancer
vincenzocalia@vincenzos-MacBook-Air helm_charts % 
</code></pre>
<p>Reading here <a href="https://learn.microsoft.com/en-us/azure/aks/faq#why-are-two-resource-groups-created-with-aks" rel="nofollow noreferrer">https://learn.microsoft.com/en-us/azure/aks/faq#why-are-two-resource-groups-created-with-aks</a></p>
<blockquote>
<p>To enable this architecture, each AKS deployment spans two resource groups:
You create the first resource group. This group contains only the Kubernetes service resource. The AKS resource provider automatically creates the second resource group during deployment. An example of the second resource group is MC_myResourceGroup_myAKSCluster_eastus. For information on how to specify the name of this second resource group, see the next section.
The second resource group, known as the node resource group, contains all of the infrastructure resources associated with the cluster. These resources include the Kubernetes node VMs, virtual networking, and storage. By default, the node resource group has a name like MC_myResourceGroup_myAKSCluster_eastus. AKS automatically deletes the node resource group whenever the cluster is deleted, so it should only be used for resources that share the cluster's lifecycle.</p>
</blockquote>
<p>Should I pass the first or the second group depending of what kind of resource I'm creating?
E.g. <code>kubernetes_service</code> needs 1st rg, while <code>azurerm_public_ip</code> needs the 2nd rg?</p>
<p>What is it that I'm missing out here?
Please explain it like I was 5 years old because I'm feeling like right now..</p>
<p>Many thanks</p>

Vincenzo
<p>Finally found what the problem was.</p>
<p>Indeed the <code>Public IP</code> needs to be created in the <code>node resource group</code>  because the ingress controller, with the <code>loadBalancerIP</code> assigned to the <code>Public IP</code> address, is going to look for it in the <code>node resource group</code> so if you create it in the <code>resource group</code> fails with the error I was getting.</p>
<p>The node resource group name is assigned at cluster creation eg. <code>MC_myResourceGroup_myAKSCluster_eastus</code>, but you can name it  as you wish using the parameter <code>node_resource_group = var.node_resource_group_name</code>.</p>
<p>Also, the Public IP <code>sku</code> <code>&quot;Standard&quot;</code> (to be specified) or <code>&quot;Basic&quot;</code> ( default), and the cluster <code>load_balancer_sku</code> <code>&quot;standard&quot;</code> or <code>&quot;basic&quot;</code>(no default value her, it needs to be specified) have to match.</p>
<p>I also put the Public IP in the cluster module so it can depend on it, to avoid being created before it and failing as the <code>node resource group</code> has not been created yet, couldn't set that dependency correctly in <code>main.tf</code> file.</p>
<p>So the working configuration is now:</p>
<h1>main</h1>
<pre><code>terraform {
  required_version = &quot;&gt;=1.1.0&quot;
  required_providers {
    azurerm = {
      source = &quot;hashicorp/azurerm&quot;
       version = &quot;~&gt; 3.0.2&quot;
    }
  }
}

provider &quot;azurerm&quot; {
  features {
    resource_group {
      prevent_deletion_if_contains_resources = false
    }
  }
  subscription_id   = var.azure_subscription_id
  tenant_id         = var.azure_subscription_tenant_id
  client_id         = var.service_principal_appid
  client_secret     = var.service_principal_password
}


provider &quot;kubernetes&quot; {
  host = &quot;${module.cluster.host}&quot;
  client_certificate = &quot;${base64decode(module.cluster.client_certificate)}&quot;
  client_key = &quot;${base64decode(module.cluster.client_key)}&quot;
  cluster_ca_certificate = &quot;${base64decode(module.cluster.cluster_ca_certificate)}&quot;
}

provider &quot;helm&quot; {
  kubernetes {
    host                   = &quot;${module.cluster.host}&quot;
    client_certificate     = &quot;${base64decode(module.cluster.client_certificate)}&quot;
    client_key             = &quot;${base64decode(module.cluster.client_key)}&quot;
    cluster_ca_certificate = &quot;${base64decode(module.cluster.cluster_ca_certificate)}&quot;
  }
}



module &quot;cluster&quot; {
  source = &quot;./modules/cluster&quot;
  location = var.location
  vm_size = var.vm_size
  resource_group_name = var.resource_group_name
  node_resource_group_name = var.node_resource_group_name
  kubernetes_version = var.kubernetes_version
  ssh_key = var.ssh_key
  sp_client_id = var.service_principal_appid
  sp_client_secret = var.service_principal_password
}




module &quot;ingress-controller&quot; {
  source = &quot;./modules/ingress-controller&quot;
  public_ip_address = module.cluster.public_ip_address
  depends_on = [
    module.cluster.public_ip_address
  ]
}
</code></pre>
<h1>cluster</h1>
<pre><code>resource &quot;azurerm_resource_group&quot; &quot;resource_group&quot; {
  name     = var.resource_group_name
  location = var.location
    tags = {
    Environment = &quot;test&quot;
    Team = &quot;DevOps&quot;
  }
}
resource &quot;azurerm_kubernetes_cluster&quot; &quot;server_cluster&quot; {
  name                = &quot;server_cluster&quot;
  ### choose the resource goup to use for the cluster
  location            = azurerm_resource_group.resource_group.location
  resource_group_name = azurerm_resource_group.resource_group.name
  ### decide the name of the cluster &quot;node&quot; resource group, if unset will be named automatically 
  node_resource_group = var.node_resource_group_name
  dns_prefix          = &quot;fixit&quot;
  kubernetes_version = var.kubernetes_version
  # sku_tier = &quot;Paid&quot;

  default_node_pool {
    name       = &quot;default&quot;
    node_count = 1
    min_count = 1
    max_count = 3
    vm_size = var.vm_size

    type = &quot;VirtualMachineScaleSets&quot;
    enable_auto_scaling = true
    enable_host_encryption = false
    # os_disk_size_gb = 30
  }

  service_principal {
    client_id = var.sp_client_id
    client_secret = var.sp_client_secret
  }

  tags = {
    Environment = &quot;Production&quot;
  }

  linux_profile {
    admin_username = &quot;azureuser&quot;
    ssh_key {
        key_data = var.ssh_key
    }
  }
  network_profile {
      network_plugin = &quot;kubenet&quot;
      load_balancer_sku = &quot;basic&quot; 
    
  }
  http_application_routing_enabled = false
  depends_on = [
    azurerm_resource_group.resource_group
  ]
}

resource &quot;azurerm_public_ip&quot; &quot;public-ip&quot; {
  name                = &quot;fixit-public-ip&quot;
  location            = var.location
  # resource_group_name = var.resource_group_name
  resource_group_name = var.node_resource_group_name
  allocation_method   = &quot;Static&quot;
  domain_name_label = &quot;fixit&quot;
  # sku = &quot;Standard&quot;

depends_on = [
  azurerm_kubernetes_cluster.server_cluster
]
}
</code></pre>
<h1>ingress controller</h1>
<pre><code>resource &quot;helm_release&quot; &quot;nginx&quot; {
  name      = &quot;ingress-nginx&quot;
  repository = &quot;ingress-nginx&quot;
  chart     = &quot;ingress-nginx/ingress-nginx&quot;
  namespace = &quot;default&quot;

  set {
    name  = &quot;controller.service.externalTrafficPolicy&quot;
    value = &quot;Local&quot;
  }

  set {
    name = &quot;controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-internal&quot;
    value = &quot;true&quot;
  }

  set {
    name  = &quot;controller.service.loadBalancerIP&quot;
    value = var.public_ip_address
  }

  set {
    name  = &quot;controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path&quot;
    value = &quot;/healthz&quot;
  }
} 
</code></pre>
<h1>ingress service</h1>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  # namespace: default
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;
    nginx.ingress.kubernetes.io/use-regex: &quot;true&quot;
    nginx.ingress.kubernetes.io/rewrite-target: /$2$3$4
spec:
  ingressClassName: nginx
  rules:
    # - host: fixit.westeurope.cloudapp.azure.com #dns from Azure PublicIP


### Node.js server
  - http:
      paths:
      - path: /(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: server-clusterip-service
            port:
              number: 80 

  - http:
      paths:
      - path: /server(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: server-clusterip-service
            port:
              number: 80
...

other services omitted
</code></pre>
<p>Hope this can help others having difficulties in getting the setup right.
Cheers.</p>
