<p>I'm trying to start a simple embedded hazelcast server in a spring boot application on kubernetes. 
I'm using hazelcast-kubernetes plugin and have followed the documentation <a href="https://github.com/hazelcast/hazelcast-kubernetes" rel="nofollow noreferrer">here</a>.
I tried both LoadBalancer and NodePort service types.</p>

<p>I also tried to deploy the sample application using <a href="https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/kubernetes/samples/embedded" rel="nofollow noreferrer">this example</a> (using the same docker image) and tried both LoadBalancer and NodePort service types.</p>

<p>In both cases I get the below error after 10 retries to connect to Kubernetes Master.
Where a.b.c.d is the IP address of the kubernetes master node.</p>

<pre><code>2020-03-09 18:20:57.322  INFO 1 --- [           main] c.h.c.LifecycleService                   : [a.b.c.d]:5701 [dev] [3.12] [a.b.c.d]:5701 is STARTING
2020-03-09 18:20:57.820  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [1] retrying in 1 seconds...
2020-03-09 18:20:59.339  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [2] retrying in 2 seconds...
2020-03-09 18:21:01.713  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [3] retrying in 3 seconds...
2020-03-09 18:21:05.113  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [4] retrying in 5 seconds...
2020-03-09 18:21:10.198  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [5] retrying in 7 seconds...
2020-03-09 18:21:17.807  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [6] retrying in 11 seconds...
2020-03-09 18:21:49.216  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [7] retrying in 17 seconds...
2020-03-09 18:22:06.319  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [8] retrying in 25 seconds...
2020-03-09 18:22:31.963  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [9] retrying in 38 seconds...
2020-03-09 18:23:15.423  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [10] retrying in 57 seconds...
2020-03-09 18:24:23.099 ERROR 1 --- [           main] c.h.i.c.i.DiscoveryJoiner                : [a.b.c.d]:5701 [dev] [3.12] Failure in KubernetesClient
com.hazelcast.kubernetes.KubernetesClientException: Failure in KubernetesClient
        at com.hazelcast.kubernetes.DefaultKubernetesClient.callGet(DefaultKubernetesClient.java:109) ~[hazelcast-kubernetes-1.3.1.jar!/:?]
        at com.hazelcast.kubernetes.DefaultKubernetesClient.endpointsByName(DefaultKubernetesClient.java:88) ~[hazelcast-kubernetes-1.3.1.jar!/:?]
        .
        .
        .       
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [main.jar:?]
Caused by: java.net.UnknownHostException: kubernetes.default.svc
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[?:1.8.0_111]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_111]
</code></pre>

<p>In the second case, where I'm using leszko's <a href="https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/kubernetes/samples/embedded" rel="nofollow noreferrer">sample</a>, the ip address it tries to connect to is the IP of the POD.</p>

<blockquote>
  <p>[190.168.1.194]:5701 [dev] [3.11] Failure in KubernetesClient</p>
</blockquote>

<p><em>190.168.1.194 is the POD's IP address</em></p>

<p>I added the below env to the deployment yaml, but doesn't look like it is overriding this varaible.</p>

<pre><code>env:
           - name: service-name
             value: "hazelcast-embedded"

Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: null, service-port: 0, service-label: null, service-label-value: true, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}
</code></pre>

<p>You seem to have an issue with your Kubernetes cluster itself.</p>

<ol>
<li><p>First you try to use Hazelcast Kubernetes Discovery strategy <a href="https://github.com/hazelcast/hazelcast-kubernetes#kubernetes-api" rel="nofollow noreferrer">Kubernetes API</a>, which makes a REST call to Kubernetes master. Kubernetes master should be always accessible from your POD by the hostname <code>kubernetes.default.svc</code>. In your case, it's not found.</p></li>
<li><p>Then, you try to use Hazelcast Kubernetes Discovery strategy <a href="https://github.com/hazelcast/hazelcast-kubernetes#dns-lookup" rel="nofollow noreferrer">DNS Lookup</a>, which uses service DNS to resolve Hazelcast members. In your case, as mentioned by @Mesut, it's not resolving correctly the DNS of a service.</p></li>
</ol>
