<p>I'm getting my feet wet with Kubernetes, and I'm using AKS and Docker Hub.  I have a REST application set up through Docker Compose that consists of nine different microservices talking with each other.  Now I'm trying to deploy this to AKS, because in terms of going from Docker Compose to Kubernetes, there doesn't seem to be a straightforward way to connect the services through a single Minikube instance.</p>

<p>That said, I need the services to be able to call each other through REST, which led me to trying to get static IPs set up on AKS.  (By now, I could've overcomplicated things, but am still getting my feet wet.)  That led me to these two commands:</p>

<pre><code>1. az network public-ip create --resource-group &lt;resourceGroup&gt; --name &lt;sameAsValueOf_resourceGroup&gt;
   --allocation-method static

2. az network public-ip list --resource-group &lt;resourceGroup&gt; --query [0].ipAddress --output tsv
</code></pre>

<p>Supposedly, the <code>list</code> command should return the same thing as the <code>create</code> command, but in my case, it doesn't.  For example, <code>create</code> might show <code>10.x.x.x</code> as the value of <code>publicIp.ipAddress</code>, but immediately thereafter, <code>list</code> might return <code>52.y.y.y</code>.</p>

<p>This is totally inconsistent with Microsoft's documentation, and when I try to include either IP in my k8s yaml as follows, it leads to <code>EXTERNAL-IP</code> getting stuck at <code>&lt;Pending&gt;</code>:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: warehouse-microservice
spec:
  type: LoadBalancer
  loadBalancerIP: 10.x.x.x
  ports:
  - port: 80
  selector:
    app: warehouse-microservice
</code></pre>

<p>In the above yaml, including <code>spec.ports[0].nodePort</code> apparently breaks things altogether.  However if I take out <code>spec.loadBalancerIP</code> and <em>then</em> add <code>spec.ports[0].nodePort</code>, everything suddenly works fine, <strong><em>and</em></strong> I can access the service through an external IP; it's just that the IP is randomly chosen.</p>

<p>So my question is: <em>Why would <code>create</code> and <code>list</code> show two different IPs, despite MS's documentation, and how can I get it to work well enough to assign a static external IP?</em></p>

Panzercrisis
<p>When creating a new AKS cluster, behind the scenes multiple network resources created ( inside <code>MC_${RESOURCE_GROUP_NAME}${CLUSTER_NAME}${LOCATION}</code> resource group). One of the resources is a static public IP (similar to the one you create using the Azure CLI command). You can find this static IP inside <code>MC resource group =&gt; Load balancer =&gt; Frontend IP configuration</code>.</p>

<p>The issue is with <code>az network public-ip list --resource-group &lt;resourceGroup&gt; --query [0].ipAddress --output tsv</code> command.</p>

<p>this command returns the first Static Ip under a given resource group. since the <code>--query</code> pointing to the first object inside the array -  <code>[0].ipAddress</code> you always get the same IP address, which is the IP address that created automatically.</p>

<p>But, Even if you chose the right IP you still face an issue (&lt; pending >) since the <a href="https://learn.microsoft.com/en-us/azure/aks/egress" rel="nofollow noreferrer">documentation</a> mentions that - <code>When multiple addresses are configured on the Azure Load Balancer, egress uses the first IP on that load balancer (Frontend IP Configuration)</code>.</p>

<p><strong>Therefore, we must use the first static IP created automatically when creating the cluster.</strong> </p>

<p><strong>End to end solution that worked for me:</strong></p>

<ol>
<li>Get the static public IP address: (You don't need to create a new one)</li>
</ol>

<pre class="lang-sh prettyprint-override"><code>EGRESS_IP=$(az network public-ip list --resource-group MC_${RESOURCE_GROUP_NAME}_${CLUSTER_NAME}_${LOCATION} --query '[0].ipAddress' --output tsv)
echo The egress ip address is: $EGRESS_IP
</code></pre>

<ol start="2">
<li>Create a service with the static IP:</li>
</ol>

<pre class="lang-sh prettyprint-override"><code> cat &lt;&lt; EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: azure-egress
  namespace: ${NAMESPACE}
spec:
  loadBalancerIP: ${EGRESS_IP}
  type: LoadBalancer
  ports:
  - port: 80
EOF
</code></pre>

<ol start="4">
<li>Ensure that the service is up and running (check that you receive external-ip and it's not stuck, This might take 1-3 minutes):</li>
</ol>

<pre class="lang-sh prettyprint-override"><code>kubectl get svc -n ${NAMESPACE}
</code></pre>

<ol start="5">
<li>To verify that the egress service is working as expected we will use a DNS look-up service:</li>
</ol>

<pre class="lang-sh prettyprint-override"><code>kubectl run -it --rm aks-egress-ip-check -n ${NAMESPACE}--image=debian --generator=run-pod/v1
</code></pre>

<ol start="6">
<li>Run the test:</li>
</ol>

<pre class="lang-sh prettyprint-override"><code>apt-get update &amp;&amp; \
apt-get install curl -y &amp;&amp; \
curl -s checkip.dyndns.org
</code></pre>
