<p>I got the following setup:</p>
<blockquote>
<p>Azure Kubernetes -&gt; Ingress-Nginx-Controller (uses Azure Load-Balancer) -&gt; External DNS</p>
</blockquote>
<p>I am exposing the Ingress-Nginx-Controller via an Ingress, backed by the Azure Load Balancer Controller using public IP.</p>
<p>I have used the following terraform code to setup the Azure Kubernetes</p>
<pre><code>resource &quot;azurerm_virtual_network&quot; &quot;test&quot; {
  name                = var.virtual_network_name
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  address_space       = [var.virtual_network_address_prefix]

  subnet {
    name           = var.aks_subnet_name
    address_prefix = var.aks_subnet_address_prefix
  }

  tags = var.tags
}

data &quot;azurerm_subnet&quot; &quot;kubesubnet&quot; {
  name                 = var.aks_subnet_name
  virtual_network_name = azurerm_virtual_network.test.name
  resource_group_name  = azurerm_resource_group.rg.name
  depends_on           = [azurerm_virtual_network.test]
}

resource &quot;azurerm_user_assigned_identity&quot; &quot;aks_external_dns_managed_id&quot; {
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location

  name = &quot;aks-external-dns-managed-id&quot;

  depends_on = [
    azurerm_resource_group.rg
  ]
}

resource &quot;azurerm_kubernetes_cluster&quot; &quot;k8s&quot; {
  name       = var.aks_name
  location   = azurerm_resource_group.rg.location
  dns_prefix = var.aks_dns_prefix

  resource_group_name = azurerm_resource_group.rg.name

  http_application_routing_enabled = false

  linux_profile {
    admin_username = var.vm_user_name

    ssh_key {
      key_data = file(var.public_ssh_key_path)
    }
  }

  default_node_pool {
    name            = &quot;agentpool&quot;
    node_count      = var.aks_agent_count
    vm_size         = var.aks_agent_vm_size
    os_disk_size_gb = var.aks_agent_os_disk_size
    vnet_subnet_id  = data.azurerm_subnet.kubesubnet.id
  }

  identity {
    type = &quot;SystemAssigned&quot;
  }

  network_profile {
    network_plugin     = &quot;azure&quot;
    dns_service_ip     = var.aks_dns_service_ip
    docker_bridge_cidr = var.aks_docker_bridge_cidr
    service_cidr       = var.aks_service_cidr
  }

  # Enabled the cluster configuration to the Azure kubernets with RBAC
  azure_active_directory_role_based_access_control { 
    managed                     = var.azure_active_directory_role_based_access_control_managed
    admin_group_object_ids      = var.active_directory_role_based_access_control_admin_group_object_ids
    azure_rbac_enabled          = var.azure_rbac_enabled
  }

  oms_agent {
    log_analytics_workspace_id  = module.log_analytics_workspace[0].id
  }

  timeouts {
    create = &quot;20m&quot;
    delete = &quot;20m&quot;
  }  

  depends_on = [data.azurerm_subnet.kubesubnet,module.log_analytics_workspace]
  tags       = var.tags
}

resource &quot;azurerm_role_assignment&quot; &quot;ra5&quot; {
  scope                = data.azurerm_subnet.kubesubnet.id
  role_definition_name = &quot;Network Contributor&quot;
  principal_id         = azurerm_user_assigned_identity.aks_external_dns_managed_id.principal_id
  depends_on           = [azurerm_kubernetes_cluster.k8s,data.azurerm_subnet.kubesubnet]
}

resource &quot;azurerm_role_assignment&quot; &quot;ra6&quot; {
  scope                = module.container_registry.0.id
  role_definition_name = &quot;AcrPush&quot;
  principal_id         = azurerm_user_assigned_identity.aks_external_dns_managed_id.principal_id
  depends_on           = [azurerm_kubernetes_cluster.k8s,module.container_registry]
}

resource &quot;azurerm_role_assignment&quot; &quot;ra7&quot; {
  scope                = azurerm_resource_group.rg.id
  role_definition_name = &quot;Reader&quot;
  principal_id         = azurerm_user_assigned_identity.aks_external_dns_managed_id.principal_id
  depends_on           = [azurerm_kubernetes_cluster.k8s]
}

resource &quot;azurerm_role_assignment&quot; &quot;ra8&quot; {
  scope                = azurerm_dns_zone.dns_zone.id
  role_definition_name = &quot;Contributor&quot;
  principal_id         = azurerm_user_assigned_identity.aks_external_dns_managed_id.principal_id
  depends_on           = [azurerm_kubernetes_cluster.k8s,]
}
</code></pre>
<p>and following details are displayed</p>
<pre><code>output &quot;subscriptionId&quot; {
  value = &quot;${data.azurerm_client_config.current.subscription_id}&quot;
}

output &quot;resource_group_name&quot; {
  value = azurerm_resource_group.rg.name
}    

output &quot;aks_external_dns_managed_id&quot; {
  value = azurerm_user_assigned_identity.aks_external_dns_managed_id.client_id
}
</code></pre>
<p><a href="https://i.stack.imgur.com/c8IqG.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/c8IqG.png" alt="enter image description here" /></a></p>
<p>and added the user managed identity as a user assigned identity in the AKS VMSS</p>
<p><a href="https://i.stack.imgur.com/b1BNU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/b1BNU.png" alt="enter image description here" /></a></p>
<p>and executed the following commands</p>
<p>Install Nginx Ingress Controller</p>
<pre><code>kubectl create namespace ingress-basic
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx  --namespace ingress-basic 
</code></pre>
<p><a href="https://i.stack.imgur.com/9PV0D.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9PV0D.png" alt="enter image description here" /></a></p>
<p>Update the details in the <strong>azure.json</strong></p>
<pre><code>{
  &quot;tenantId&quot;: &quot;tenant ID GUID&quot;,
  &quot;subscriptionId&quot;: &quot;subscription ID GUID&quot;,
  &quot;resourceGroup&quot;: &quot;rg-blessed-baboon&quot;, 
  &quot;useManagedIdentityExtension&quot;: true,
  &quot;userAssignedIdentityID&quot;: &quot;1a8e6a4b-2d84-447d-8be5-efd1e95e2dda&quot;
}
</code></pre>
<p>and created the secret</p>
<pre><code>kubectl create secret generic azure-config-file --from-file=azure.json
</code></pre>
<p>and applied the</p>
<pre><code>kubectl apply -f external-dns.yml
</code></pre>
<p>external-dns.yml</p>
<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [&quot;&quot;]
  resources: [&quot;services&quot;,&quot;endpoints&quot;,&quot;pods&quot;, &quot;nodes&quot;]
  verbs: [&quot;get&quot;,&quot;watch&quot;,&quot;list&quot;]
- apiGroups: [&quot;extensions&quot;,&quot;networking.k8s.io&quot;]
  resources: [&quot;ingresses&quot;] 
  verbs: [&quot;get&quot;,&quot;watch&quot;,&quot;list&quot;]
- apiGroups: [&quot;&quot;]
  resources: [&quot;nodes&quot;]
  verbs: [&quot;list&quot;]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: k8s.gcr.io/external-dns/external-dns:v0.13.2
        args:
        - --source=service
        - --source=ingress
        #- --domain-filter=example.com # (optional) limit to only example.com domains; change to match the zone created above.
        - --provider=azure
        #- --azure-resource-group=externaldns # (optional) use the DNS zones from the specific resource group
        volumeMounts:
        - name: azure-config-file
          mountPath: /etc/kubernetes
          readOnly: true
      volumes:
      - name: azure-config-file
        secret:
          secretName: azure-config-file
</code></pre>
<p>and Deployed the application</p>
<p>kubectl apply -f .\deployment.yaml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1-nginx-deployment
  labels:
    app: app1-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app1-nginx
  template:
    metadata:
      labels:
        app: app1-nginx
    spec:
      containers:
        - name: app1-nginx
          image: stacksimplify/kube-nginxapp1:1.0.0
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: app1-nginx-clusterip-service
  labels:
    app: app1-nginx
spec:
  type: ClusterIP
  selector:
    app: app1-nginx
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginxapp1-ingress-service
  annotations:
    kubernetes.io/ingress.class: &quot;nginx&quot;
spec:
  rules:
    - host: eapp1.eat-eggs.ca
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app1-nginx-clusterip-service
                port: 
                  number: 80
---
</code></pre>
<p>It got deployed and I can access the application using cluster IP</p>
<p><a href="https://i.stack.imgur.com/Q1HcA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Q1HcA.png" alt="enter image description here" /></a></p>
<p>DNS did not get updated even after 20 mins after the deployment.</p>
<p>External DNS throws the following error</p>
<pre><code>kubectl logs -f external-dns-765645d59d-d2bsp
</code></pre>
<p>Logs:</p>
<blockquote>
<p>time=&quot;2023-02-11T02:28:30Z&quot; level=error
msg=&quot;azure.BearerAuthorizer#WithAuthorization: Failed to refresh the
Token for request to
<a href="https://management.azure.com/subscriptions/" rel="nofollow noreferrer">https://management.azure.com/subscriptions/</a>/resourceGroups/rg-blessed-baboon/providers/Microsoft.Network/dnsZones?api-version=2018-05-01:
StatusCode=404 -- Original Error: adal: Refresh request failed. Status
Code = '404'. Response body: clientID in request: 1a8e##### REDACTED
#####2dda, getting assigned identities for pod default/external-dns-765645d59d-d2bsp in CREATED state failed after 16
attempts, retry duration [5]s, error: . Check MIC pod logs for
identity assignment errors\n Endpoint
<a href="http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&amp;client_id=1a8e6a4b-2d84-447d-8be5-efd1e95e2dda&amp;resource=https%3A%2F%2Fmanagement.core.windows.net%2F%22" rel="nofollow noreferrer">http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&amp;client_id=1a8e6a4b-2d84-447d-8be5-efd1e95e2dda&amp;resource=https%3A%2F%2Fmanagement.core.windows.net%2F&quot;</a></p>
</blockquote>
<p>What am I missing?</p>
<p><strong>Update#1</strong></p>
<p>I have replaced the external DNS file with the below</p>
<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
  - apiGroups: [&quot;&quot;]
    resources: [&quot;services&quot;,&quot;endpoints&quot;,&quot;pods&quot;, &quot;nodes&quot;]
    verbs: [&quot;get&quot;,&quot;watch&quot;,&quot;list&quot;]
  - apiGroups: [&quot;extensions&quot;,&quot;networking.k8s.io&quot;]
    resources: [&quot;ingresses&quot;]
    verbs: [&quot;get&quot;,&quot;watch&quot;,&quot;list&quot;]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
  - kind: ServiceAccount
    name: external-dns
    namespace: ingress-basic
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
        - name: external-dns
          image: registry.k8s.io/external-dns/external-dns:v0.13.2
          args:
            - --source=service
            - --source=ingress
            #- --domain-filter=example.com # (optional) limit to only example.com domains; change to match the zone created above.
            - --provider=azure
            #- --azure-resource-group=MyDnsResourceGroup # (optional) use the DNS zones from the tutorial's resource group
            #- --txt-prefix=externaldns-
          volumeMounts:
            - name: azure-config-file
              mountPath: /etc/kubernetes
              readOnly: true
      volumes:
        - name: azure-config-file
          secret:
            secretName: azure-config-file
</code></pre>
<p>and tried with both <strong>Client ID &amp; Object (principal) ID</strong>. Issue is still persist.</p>

One Developer
<p>you didnt attach managed identity created for external dns to aks nodes. normally you have <a href="https://github.com/Azure/aad-pod-identity" rel="nofollow noreferrer">aad pod identity controller</a> handling that (and security around that), but you can simply go to the aks managed group click on the vmss and assign managed identity for external dns to the nodepool</p>
