<p>I am creating an Amazon EKS cluster using jenkins-x with:</p>

<pre><code>jx create cluster eks -n demo --node-type=t3.xlarge --nodes=1 --nodes-max=5 --nodes-min=1 --skip-installation
</code></pre>

<p>After that, I add the cluster-autoscaler IAM policy for auto discovery and the added tags on the autoscaling group and the created instance, according <a href="https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws" rel="nofollow noreferrer">this</a> guide.</p>

<p>I add the rbac roles for tiller and the autoscaler with this file (<code>kubectl create -f rbac-config.yaml</code>):</p>

<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: autoscaler
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: autoscaler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: autoscaler
    namespace: kube-system
</code></pre>

<p>I installed tiller:</p>

<p><code>helm init --service-account tiller</code></p>

<p>and installed the cluster autoscaler:</p>

<p><code>helm install stable/cluster-autoscaler -f cluster-autoscaler-values.yaml --name cluster-autoscaler --namespace kube-system</code></p>

<p>Then I install the jenkins-x system:</p>

<p><code>jx install --provider=eks --domain=mydomain.com --default-environment-prefix=demo --skip-setup-tiller</code></p>

<p>I just accept all the defaults on the questions (nginx-ingress is created for me).</p>

<p>Then I create a default spring-boot-rest-prometheus app:</p>

<p><code>jx create quickstart</code></p>

<p>again, accepting all the defaults.
This works fine, the application is picked up by jenkins is compiled, which I can see in:</p>

<p><code>http://jenkins.jx.mydomain.com</code></p>

<p>and I can reach the app through:</p>

<p><code>http://spring-boot-rest-prometheus.jx-staging.mydomain.com</code></p>

<p>Then I run a test to see if the autoscaler is working correctly, so I open up the file in the <code>charts/spring-boot-rest-prometheus/values.yaml</code> and change <code>replicaCount: 1</code> to <code>replicaCount: 8</code>. Commit and push. This kicks of the Jenkins pipeline and spins up a new node because the autoscaler sees that there are not enough cpu resources on the first node.</p>

<p>After the second node has come up, I cannot reach Jenkins and the app anymore via the domain names. So for some reason, my ingress is not working anymore.</p>

<p>I have played around with this a lot, and manually changing the desired number of nodes directly on EC2, and when there is an even number of nodes, the domains are not reachable and when there is an odd number of nodes the domains are reachable.</p>

<p>I do not think this is related to the autoscaler, because the scale up and the scale down are working fine, and the problem is also there if I manually change the desired nodes of the server.</p>

<p>What causes the ingress to fail for an even number of nodes? How can I investigate this issue further? </p>

<p>Logs and desriptors for all ingress parts are posted <a href="https://gist.github.com/martijnburger/1e842e6c1d018044e4f01f9054f9bfb6" rel="nofollow noreferrer">here</a>.</p>

Martijn Burger
<p>You can debug this by looking at the <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html" rel="nofollow noreferrer">AWS ASG</a> (AutoScaling Group) and the <a href="https://aws.amazon.com/elasticloadbalancing/" rel="nofollow noreferrer">load balancer</a> (ELB) target instances.</p>

<p>You can see that the instances are being added to the ASG:</p>

<p><a href="https://i.stack.imgur.com/BhKbc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BhKbc.png" alt="ASG"></a></p>

<p>Then you can see in your load balancer that the instances are in service:</p>

<p><a href="https://i.stack.imgur.com/aoqxU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/aoqxU.png" alt="In service"></a></p>

<p>It could be that some of the even number of instances are not in service. Do they happen to be in a different availability zone? Are the ones that are 'odd' numbers being removed from the ELB? is traffic not being forwarded to them?</p>
