<p>I am trying to create an AKS cluster in azure using the <code>terraform</code>. My requirements are as follows:</p>
<ol>
<li><p>Create a site-to-site VPN connection where the gateway in the subnet of range <code>172.30.0.0/16</code> - <strong>This is done</strong></p>
</li>
<li><p>Install Azure AKS cluster with Azure CNI and pod's should be in the range of VPN CIDR (<code>172.30.0.0/16</code>).</p>
</li>
</ol>
<p>Here's my terraform code. I read that if you use <code>azure</code> as your <code>network_policy</code> and <code>network_plugin</code>, you can't set the <code>pod_cidr</code> - <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#network_plugin" rel="nofollow noreferrer">source</a></p>
<p><strong>Then how can I do this so my PODs can reach the on-premise network through the site-to-site vpn?</strong></p>
<pre><code>
resource &quot;azurerm_kubernetes_cluster&quot; &quot;k8s_cluster&quot; {
  lifecycle {
    ignore_changes = [
      default_node_pool[0].node_count
    ]
    prevent_destroy = false
  }

  name                = var.cluster_name
  location            = var.location
  resource_group_name = var.rg_name
  dns_prefix          = var.dns_prefix
  kubernetes_version  = var.kubernetes_version
  # node_resource_group = var.resource_group_name

  default_node_pool {
    name                  = var.default_node_pool.name
    node_count            = var.default_node_pool.node_count
    max_count             = var.default_node_pool.max_count
    min_count             = var.default_node_pool.min_count
    vm_size               = var.default_node_pool.vm_size
    os_disk_size_gb       = var.default_node_pool.os_disk_size_gb
    # vnet_subnet_id        = var.vnet_subnet_id
    max_pods              = var.default_node_pool.max_pods
    type                  = var.default_node_pool.agent_pool_type
    enable_node_public_ip = var.default_node_pool.enable_node_public_ip
    enable_auto_scaling   = var.default_node_pool.enable_auto_scaling

    tags = merge(var.common_tags)
  }

  identity {
    type = var.identity
  }

  network_profile {
    network_plugin     = var.network_plugin #azure
    network_policy     = var.network_policy #&quot;azure&quot;
    load_balancer_sku  = var.load_balancer_sku #&quot;standard&quot;
    # pod_cidr  = var.pod_cidr | When network_plugin is set to azure - the vnet_subnet_id field in the default_node_pool block must be set and pod_cidr must not be set.
  }
  

  tags = merge(var.common_tags)
}
</code></pre>
<pre><code># AKS cluster related variables
cluster_name       = &quot;test-cluster&quot;
dns_prefix         = &quot;testjana&quot;
kubernetes_version = &quot;1.22.15&quot;
default_node_pool = {
  name                  = &quot;masternp&quot; # for system pods
  node_count            = 1
  vm_size               = &quot;standard_e4bds_v5&quot; # 4 vcpu and 32 Gb of memory
  enable_auto_scaling   = false
  enable_node_public_ip = false
  min_count             = null
  max_count             = null
  max_pods              = 100
  os_disk_size_gb       = 80
  agent_pool_type       = &quot;VirtualMachineScaleSets&quot;
}
admin_username    = &quot;jananathadmin&quot;
ssh_public_key    = &quot;public_key&quot;
identity          = &quot;SystemAssigned&quot;
network_plugin    = &quot;azure&quot;
network_policy    = &quot;azure&quot;
load_balancer_sku = &quot;standard&quot;

</code></pre>

Jananath Banuka
<p>Default, all PODs in AKS will communicate each other, when we want to restrict the traffic, network policies can be used to allow or deny traffic between pods.</p>
<p>Here is the tutorial <a href="https://learn.microsoft.com/en-us/azure/vpn-gateway/tutorial-site-to-site-portal" rel="nofollow noreferrer">link</a></p>
<p>Reproduced the same via terraform using below code snippet to connect cluster with azure CNI and a vnet gateway which links our on-prem environment to azure via a site-to-site VPN.</p>
<p><strong>Step1:</strong>
<strong>main tf file as follows</strong></p>
<pre><code>resource &quot;azurerm_resource_group&quot; &quot;example&quot; {
  name     = &quot;*****-****&quot;
  location = &quot;East US&quot;
}

resource &quot;azurerm_role_assignment&quot; &quot;role_acrpull&quot; {
  scope                            = azurerm_container_registry.acr.id
  role_definition_name             = &quot;AcrPull&quot;
  principal_id                     = azurerm_kubernetes_cluster.demo.kubelet_identity.0.object_id
}

resource &quot;azurerm_container_registry&quot; &quot;acr&quot; {
  name                = &quot;acrswarna&quot;
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  sku                 = &quot;Standard&quot;
  admin_enabled       = false
}
resource &quot;azurerm_virtual_network&quot; &quot;puvnet&quot; {
    name                = &quot;Publics_VNET&quot;
   resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  address_space       = [&quot;10.19.0.0/16&quot;]
}
resource &quot;azurerm_subnet&quot; &quot;example&quot; {
  name                 = &quot;GatewaySubnet&quot;
   resource_group_name = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.puvnet.name
  address_prefixes     = [&quot;10.19.3.0/24&quot;]
}
resource &quot;azurerm_subnet&quot; &quot;osubnet&quot; {
  name                 = &quot;Outer_Subnet&quot;
  resource_group_name = azurerm_resource_group.example.name
  address_prefixes     = [&quot;10.19.1.0/24&quot;]
  virtual_network_name = azurerm_virtual_network.puvnet.name
}

resource &quot;azurerm_kubernetes_cluster&quot; &quot;demo&quot; {
  name                = &quot;demo-aksnew&quot;
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = &quot;demo-aks&quot;

  default_node_pool {
    name                = &quot;default&quot;
    node_count          = 2
    vm_size             = &quot;standard_e4bds_v5&quot;
    type                = &quot;VirtualMachineScaleSets&quot;
    enable_auto_scaling = false
    min_count             = null
    max_count             = null
    max_pods              = 100
    //vnet_subnet_id = azurerm_subnet.osubnet.id
  }
  
  identity {
    type = &quot;SystemAssigned&quot;
  }

  network_profile {
    network_plugin    = &quot;azure&quot;
    load_balancer_sku = &quot;standard&quot;
    network_policy    = &quot;azure&quot;
  }

  tags = {
    Environment = &quot;Development&quot;
  }
}
resource &quot;azurerm_public_ip&quot; &quot;example&quot; {
  name                = &quot;pips-firewall&quot;
   resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
   allocation_method = &quot;Static&quot;
  sku                 = &quot;Standard&quot;
}
resource &quot;azurerm_virtual_network_gateway&quot; &quot;example&quot; {
  name                = &quot;test&quot;
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name

  type     = &quot;Vpn&quot;
  vpn_type = &quot;RouteBased&quot;

  active_active = false
  enable_bgp    = false
  sku           = &quot;VpnGw1&quot;

  ip_configuration {
    name                          = &quot;vnetGatewayConfig&quot;
    public_ip_address_id          = azurerm_public_ip.example.id
    private_ip_address_allocation = &quot;Dynamic&quot;
    subnet_id                     = azurerm_subnet.example.id
  }

  vpn_client_configuration {
    address_space = [&quot;172.30.0.0/16&quot;]

    root_certificate {
      name = &quot;******-****-ID-Root-CA&quot;

      public_cert_data = &lt;&lt;EOF
   **Use certificate here**
   EOF
    }
    revoked_certificate {
      name       = &quot;*****-Global-Root-CA&quot;
      thumbprint = &quot;****************&quot;
    }
  }
}
</code></pre>
<p>NOTE: Update root certificate configuration by own on above code.</p>
<p><strong>Provider tf file as follow</strong></p>
<pre><code>terraform {
  required_providers {
    azurerm = {
      source  = &quot;hashicorp/azurerm&quot;
      version = &quot;=3.0.0&quot;
    }
  }

}
provider &quot;azurerm&quot; {
  features {}
  
  skip_provider_registration = true
}
</code></pre>
<p>upon running</p>
<pre><code> terraform plan 
 terraform apply -auto-approve
</code></pre>
<p><img src="https://i.stack.imgur.com/2fqKs.png" alt="enter image description here" /></p>
<p><img src="https://i.stack.imgur.com/42Cf1.png" alt="enter image description here" /></p>
<p><img src="https://i.stack.imgur.com/tDSEE.png" alt="enter image description here" /></p>
<p>Vnet and SubNet  configurations
<img src="https://i.stack.imgur.com/MIwEg.png" alt="enter image description here" /></p>
<p>Virtual Network Gateway configuraiton as follows.
<img src="https://i.stack.imgur.com/4KgP7.png" alt="enter image description here" /></p>
<p>Deployed sample pods on cluster
<img src="https://i.stack.imgur.com/gJSjP.png" alt="enter image description here" /></p>
