{"action":"opened","issue":{"url":"https://api.github.com/repos/Netflix/eureka/issues/1134","repository_url":"https://api.github.com/repos/Netflix/eureka","labels_url":"https://api.github.com/repos/Netflix/eureka/issues/1134/labels{/name}","comments_url":"https://api.github.com/repos/Netflix/eureka/issues/1134/comments","events_url":"https://api.github.com/repos/Netflix/eureka/issues/1134/events","html_url":"https://github.com/Netflix/eureka/issues/1134","id":370236935,"node_id":"MDU6SXNzdWUzNzAyMzY5MzU=","number":1134,"title":"Unavailable Replicas When Running in AWS Using DNS Replication","user":{"login":"jawitthuhn","id":28634003,"node_id":"MDQ6VXNlcjI4NjM0MDAz","avatar_url":"https://avatars2.githubusercontent.com/u/28634003?v=4","gravatar_id":"","url":"https://api.github.com/users/jawitthuhn","html_url":"https://github.com/jawitthuhn","followers_url":"https://api.github.com/users/jawitthuhn/followers","following_url":"https://api.github.com/users/jawitthuhn/following{/other_user}","gists_url":"https://api.github.com/users/jawitthuhn/gists{/gist_id}","starred_url":"https://api.github.com/users/jawitthuhn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jawitthuhn/subscriptions","organizations_url":"https://api.github.com/users/jawitthuhn/orgs","repos_url":"https://api.github.com/users/jawitthuhn/repos","events_url":"https://api.github.com/users/jawitthuhn/events{/privacy}","received_events_url":"https://api.github.com/users/jawitthuhn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-15T16:11:01Z","updated_at":"2018-10-15T16:11:01Z","closed_at":null,"author_association":"NONE","body":"We are running Eureka and having issues where all replicas are showing up in Eureka as unavailable replicas. Each server shows all three servers, so it successfully finds the others, but the replicas are showing as unavailable. Also, each server also shows itself as a replica (I'm not sure if that is normal).\r\n\r\nConfiguration:\r\n\r\n- Three EC2 instances. Each with an extra network interface attached.\r\n- The extra network interface is then referenced from our DNS records.\r\n- This allows us to take down an instance and replace it, and just attach the network interface to the new instance to allow our Eureka instances to maintain the same IP address.\r\n- For the sake of my examples below, let's say the EC2 instances are:\r\n\r\n| **AZ**         | **EC2 Instance** | **Extra Network Adapter** |\r\n|------------|--------------|-----------------------|\r\n| eu-west-1a | 10.1.1.1     | 10.1.1.2              |\r\n| eu-west-1b | 10.1.2.1     | 10.1.2.2              |\r\n| eu-west-1c | 10.1.3.1     | 10.1.3.2              |\r\n\r\nI believe the DNS record is working fine, since it is able to read the IP addresses of the other Eureka instances without issue.\r\n\r\nThe replics show up in \"unavailable-replicas\" on the Eureka page in this format:\r\n`http://10.1.1.2:8080/eureka/v2/,http://10.1.2.2:8080/eureka/v2/,http://10.1.3.2:8080/eureka/v2/,`\r\n(where those IP addresses are the correct IP addresses for the extra network interfaces that are referred do by our DNS records)\r\n\r\nIf I sign into one of the boxes, I am able to curl to these URLs, so I do not believe there is a network issue here. Also, application instances are successfully registering with Eureka,including all three instances of Eureka (as I'd expect, it shows them with the EC2 instance IPs of 10.1.1.1, 10.1.2.1, and 10.1.3.1).\r\n\r\nWe are running on 1.9.5, but also experienced the issue with a much older version, 1.3.1.\r\n\r\nLooking in the logs, we are seeing this output every sixty seconds:\r\n```\r\n2018-10-15 16:02:20,812 ERROR com.netflix.eureka.aws.EIPManager$EIPBindingTask:444 [Eureka-EIPBinder] [run] Could not bind to EIP\r\njava.lang.StringIndexOutOfBoundsException: String index out of range: -4\r\n        at java.lang.String.substring(String.java:1967)\r\n        at com.netflix.eureka.aws.EIPManager.getEIPsFromServiceUrls(EIPManager.java:360)\r\n        at com.netflix.eureka.aws.EIPManager.getEIPsForZoneFromDNS(EIPManager.java:394)\r\n        at com.netflix.eureka.aws.EIPManager.getCandidateEIPs(EIPManager.java:316)\r\n        at com.netflix.eureka.aws.EIPManager.isEIPBound(EIPManager.java:165)\r\n        at com.netflix.eureka.aws.EIPManager$EIPBindingTask.run(EIPManager.java:431)\r\n        at java.util.TimerThread.mainLoop(Timer.java:555)\r\n        at java.util.TimerThread.run(Timer.java:505)\r\n```\r\n(this is interesting because I don't believe we are even using EIP based configuration, right?)\r\n\r\nWe also see this every five minutes (with the correct DNS name):\r\n`2018-10-15 16:01:35,083 INFO  com.netflix.discovery.shared.resolver.aws.ConfigClusterResolver:39 [AsyncResolver-bootstrap-executor-0] [getClusterEndpoints] Resolving eureka endpoints via DNS: txt.eu-west-1.eureka-qa.cloud.ourcompany.com`\r\n\r\nWhen we start our Eureka server, we are passing in system properties like this:\r\n`-Deureka.environment=qa -Deureka.datacenter=cloud -Deureka.shouldUseDns=true -Deureka.eurekaServer.domainName=eureka-qa.cloud.ourcompany.com -Deureka.eurekaServer.context=eureka/v2 -Deureka.enableSelfPreservation=false -Deureka.datacenter=cloud -Deureka.region=eu-west-1`\r\n\r\nAre we doing something wrong with our configuration? The StringIndexOutOfBoundsException issue almost looks like a bug with Eureka. We've spent a few days trying to solve this ourselves, but haven't made much progress.\r\n\r\nI'd appreciate any help anyone can give!\r\n\r\nThanks,\r\nJoseph Witthuhn"}}
{"actor":{"display_login":"jawitthuhn"}}