<p>I have created kops cluster and elasticsearch logging as below.</p>

<pre><code>kops create cluster --zones ap-southeast-1a,ap-southeast-1b,ap-southeast-1c --topology private --networking calico --master-size t2.micro --master-count 3 --node-size t2.micro --node-count 2 --cloud-labels "Project=Kubernetes,Team=Devops" ${NAME} --ssh-public-key /root/.ssh/id_rsa.pub --yes
</code></pre>

<p><a href="https://github.com/kubernetes/kops/blob/master/addons/logging-elasticsearch/v1.7.0.yaml" rel="nofollow noreferrer">https://github.com/kubernetes/kops/blob/master/addons/logging-elasticsearch/v1.7.0.yaml</a></p>

<p>then some important cluster information.</p>

<pre><code>root@ubuntu:~# kubectl get services -n kube-system
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
elasticsearch-logging   ClusterIP   100.67.69.222    &lt;none&gt;        9200/TCP        2m
kibana-logging          ClusterIP   100.67.182.172   &lt;none&gt;        5601/TCP        2m
kube-dns                ClusterIP   100.64.0.10      &lt;none&gt;        53/UDP,53/TCP   6m

root@ubuntu:~# kubectl cluster-info
Kubernetes master is running at https://${NAME}
Elasticsearch is running at https://${NAME}/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy
Kibana is running at https://${NAME}/api/v1/namespaces/kube-system/services/kibana-logging/proxy
KubeDNS is running at https://${NAME}/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
</code></pre>

<p>now when  i access "https://${NAME}/api/v1/namespaces/kube-system/services/kibana-logging"<br>
i got as below in browser</p>

<pre><code>{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "kibana-logging",
    "namespace": "kube-system",
    "selfLink": "/api/v1/namespaces/kube-system/services/kibana-logging",
    "uid": "8fc914f7-3f65-11e9-a970-0aaac13c99b2",
    "resourceVersion": "923",
    "creationTimestamp": "2019-03-05T16:41:44Z",
    "labels": {
      "k8s-addon": "logging-elasticsearch.addons.k8s.io",
      "k8s-app": "kibana-logging",
      "kubernetes.io/cluster-service": "true",
      "kubernetes.io/name": "Kibana"
    }
  },
  "spec": {
    "ports": [
      {
        "protocol": "TCP",
        "port": 5601,
        "targetPort": "ui"
      }
    ],
    "selector": {
      "k8s-app": "kibana-logging"
    },
    "clusterIP": "100.67.182.172",
    "type": "ClusterIP",
    "sessionAffinity": "None"
  },
  "status": {
    "loadBalancer": {

    }
  }
}
</code></pre>

<p>when i access "https://${NAME}/api/v1/namespaces/kube-system/services/elasticsearch-logging"</p>

<p>i got as below in browser</p>

<pre><code>{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "elasticsearch-logging",
    "namespace": "kube-system",
    "selfLink": "/api/v1/namespaces/kube-system/services/elasticsearch-logging",
    "uid": "8f7cc654-3f65-11e9-a970-0aaac13c99b2",
    "resourceVersion": "902",
    "creationTimestamp": "2019-03-05T16:41:44Z",
    "labels": {
      "k8s-addon": "logging-elasticsearch.addons.k8s.io",
      "k8s-app": "elasticsearch-logging",
      "kubernetes.io/cluster-service": "true",
      "kubernetes.io/name": "Elasticsearch"
    }
  },
  "spec": {
    "ports": [
      {
        "protocol": "TCP",
        "port": 9200,
        "targetPort": "db"
      }
    ],
    "selector": {
      "k8s-app": "elasticsearch-logging"
    },
    "clusterIP": "100.67.69.222",
    "type": "ClusterIP",
    "sessionAffinity": "None"
  },
  "status": {
    "loadBalancer": {

    }
  }
}
</code></pre>

<p>when i access "https://${NAME}/api/v1/namespaces/kubesystem/services/elasticsearch-logging/proxy/"<br>
i got below in browser</p>

<pre><code>{
  "name" : "elasticsearch-logging-0",
  "cluster_name" : "kubernetes-logging",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "5.6.4",
    "build_hash" : "8bbedf5",
    "build_date" : "2017-10-31T18:55:38.105Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
</code></pre>

<p>when i access "https://${NAME}/api/v1/namespaces/kube-system/services/kibana-logging/proxy"</p>

<p>i got error as below.</p>

<pre><code>:Error: 'dial tcp 100.111.147.69:5601: connect: connection refused'
Trying to reach: 'http://100.111.147.69:5601/'
</code></pre>

<p>why i am not getting GUI of kibana here?</p>

<p>After one hour, i got kibana logs as below</p>

<pre><code>{"type":"log","@timestamp":"2019-03-22T12:46:58Z","tags":["info","optimize"],"pid":1,"message":"Optimizing and caching bundles for graph, ml, kibana, stateSessionStorageRedirect, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2019-03-22T13:18:19Z","tags":["info","optimize"],"pid":1,"message":"Optimization of bundles for graph, ml, kibana, stateSessionStorageRedirect, timelion and status_page complete in 1880.89 seconds"}
{"type":"log","@timestamp":"2019-03-22T13:18:20Z","tags":["status","plugin:kibana@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:18:21Z","tags":["status","plugin:elasticsearch@5.6.4","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:18:21Z","tags":["status","plugin:xpack_main@5.6.4","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:18:22Z","tags":["status","plugin:graph@5.6.4","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:18:30Z","tags":["reporting","warning"],"pid":1,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2019-03-22T13:18:30Z","tags":["status","plugin:reporting@5.6.4","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:18:36Z","tags":["status","plugin:xpack_main@5.6.4","info"],"pid":1,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-22T13:18:36Z","tags":["status","plugin:graph@5.6.4","info"],"pid":1,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-22T13:18:36Z","tags":["status","plugin:reporting@5.6.4","info"],"pid":1,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-22T13:18:36Z","tags":["status","plugin:elasticsearch@5.6.4","info"],"pid":1,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-22T13:18:55Z","tags":["status","plugin:elasticsearch@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}
{"type":"log","@timestamp":"2019-03-22T13:18:57Z","tags":["license","info","xpack"],"pid":1,"message":"Imported license information from Elasticsearch for [data] cluster: mode: trial | status: active | expiry date: 2019-04-21T11:47:30+00:00"}
{"type":"log","@timestamp":"2019-03-22T13:18:57Z","tags":["status","plugin:xpack_main@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}
{"type":"log","@timestamp":"2019-03-22T13:18:57Z","tags":["status","plugin:graph@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}
{"type":"log","@timestamp":"2019-03-22T13:18:57Z","tags":["status","plugin:reporting@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}
{"type":"log","@timestamp":"2019-03-22T13:20:37Z","tags":["status","plugin:searchprofiler@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:20:37Z","tags":["status","plugin:ml@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:20:38Z","tags":["status","plugin:ml@5.6.4","info"],"pid":1,"state":"yellow","message":"Status changed from green to yellow - Waiting for Elasticsearch","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-22T13:20:38Z","tags":["status","plugin:ml@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-22T13:20:38Z","tags":["status","plugin:tilemap@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:20:38Z","tags":["status","plugin:watcher@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:20:38Z","tags":["status","plugin:grokdebugger@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:20:38Z","tags":["status","plugin:upgrade@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:20:38Z","tags":["status","plugin:console@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:20:39Z","tags":["status","plugin:metrics@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:20:41Z","tags":["status","plugin:timelion@5.6.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-22T13:20:41Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0:5601"}
{"type":"log","@timestamp":"2019-03-22T13:20:41Z","tags":["status","ui settings","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"response","@timestamp":"2019-03-22T13:26:01Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/","method":"get","headers":{"host":"MYHOSTNAME","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","cache-control":"max-age=0","upgrade-insecure-requests":"1","x-forwarded-for":"172.20.1.246","x-forwarded-uri":"/api/v1/namespaces/kube-system/services/kibana-logging/proxy/"},"remoteAddress":"100.124.142.0","userAgent":"100.124.142.0"},"res":{"statusCode":200,"responseTime":622,"contentLength":9},"message":"GET / 200 622ms - 9.0B"}
</code></pre>

<p>also curl from master node</p>

<pre><code>admin@ip-172-20-51-6:~$ curl 100.66.205.174:5601
&lt;script&gt;var hashRoute = '/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana';
var defaultRoute = '/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana';

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
</code></pre>

<p>browser log
<a href="https://i.stack.imgur.com/uKlUU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uKlUU.png" alt="enter image description here"></a></p>

k''
<p>This is happening because it takes some time to find optimal setup of ELK.
If you check log of <code>kibana-logging</code> container, you will see these output:</p>

<pre><code>$ kubectl -n kube-system logs -f kibana-logging-8675b4ffd-75rzd
{"type":"log","@timestamp":"2019-03-21T09:11:30Z","tags":["info","optimize"],"pid":1,"message":"Optimizing and caching bundles for graph, ml, kibana, stateSessionStorageRedirect, timelion and status_page. This may take a few minutes"}
</code></pre>

<p>You need to wait around 1h, then service will be available and you will able access kibana ui.</p>

<p>For more information see this <a href="https://github.com/pires/kubernetes-elasticsearch-cluster/issues/147" rel="nofollow noreferrer">github discussion</a> </p>
