title: Istio + Elasticsearch
body: <p>I'm playing with Istio and Elasticsearch, I have virtual box machines on my laptop</p>
  
  <p>I Installed Elastic search from this link <a href="https://github.com/pires/kubernetes-elasticsearch-cluster" rel="nofollow noreferrer">Kubernetes Elasticsearch Cluster</a></p>
  
  <p>I have a master es + data es, if I installed them without Istio, they run normally.</p>
  
  <p>If I inject them with Istio the data nodes can not communicate with master (it does not find it).</p>
  
  <pre><code>root@node1:/home/arkan# k get all
  NAME                             READY     STATUS    RESTARTS   AGE
  pod/es-data-6fdbcf956f-fdnc7     1/2       Running   1          1m
  pod/es-master-6b6d5fd59b-86qpb   2/2       Running   0          1m
  NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
  service/elasticsearch             NodePort    10.108.28.225   &lt;none&gt;        9200:32721/TCP   1m
  service/elasticsearch-discovery   ClusterIP   None            &lt;none&gt;        9300/TCP         1m
  service/kubernetes                ClusterIP   10.96.0.1       &lt;none&gt;        443/TCP          22h
  NAME                        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
  deployment.apps/es-data     1         1         1            0           1m
  deployment.apps/es-master   1         1         1            1           1m
  NAME                                   DESIRED   CURRENT   READY     AGE
  replicaset.apps/es-data-6fdbcf956f     1         1         0         1m
  replicaset.apps/es-master-6b6d5fd59b   1         1         1         1m
  
  
  
  
  
  root@node1:/home/arkan# k describe pod/es-data-6fdbcf956f-fdnc7
  Name:           es-data-6fdbcf956f-fdnc7
  Namespace:      default
  Node:           node2/192.168.0.214
  Start Time:     Wed, 18 Jul 2018 21:42:50 +0300
  Labels:         component=elasticsearch
                  pod-template-hash=2986795129
                  role=data
  Annotations:    sidecar.istio.io/status={"version":"55c9e544b52e1d4e45d18a58d0b34ba4b72531e45fb6d1572c77191422556ffc","initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs...
  Status:         Running
  IP:             172.16.104.22
  Controlled By:  ReplicaSet/es-data-6fdbcf956f
  Init Containers:
    init-sysctl:
      Container ID:  docker://c510035d1e823d134ad287116ef43332255758cce60cc1216ed20282b0b55e76
      Image:         busybox:1.27.2
      Image ID:      docker-pullable://busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0
      Port:          &lt;none&gt;
      Host Port:     &lt;none&gt;
      Command:
        sysctl
        -w
        vm.max_map_count=262144
      State:          Terminated
        Reason:       Completed
        Exit Code:    0
        Started:      Wed, 18 Jul 2018 21:42:52 +0300
        Finished:     Wed, 18 Jul 2018 21:42:52 +0300
      Ready:          True
      Restart Count:  0
      Environment:    &lt;none&gt;
      Mounts:
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-mkdlq (ro)
    istio-init:
      Container ID:  docker://42a8f5da07834533dcd4c26155fb344fa41edecb744e6a4c14c54c40610a450b
      Image:         docker.io/istio/proxy_init:0.8.0
      Image ID:      docker-pullable://istio/proxy_init@sha256:b0b288ee8270e054442abdd413da9395e2af39fed1792b85ec157700ef2c192f
      Port:          &lt;none&gt;
      Host Port:     &lt;none&gt;
      Args:
        -p
        15001
        -u
        1337
        -m
        REDIRECT
        -i
        *
        -x
  
        -b
        9200, 9300,
        -d
  
      State:          Terminated
        Reason:       Completed
        Exit Code:    0
        Started:      Wed, 18 Jul 2018 21:42:53 +0300
        Finished:     Wed, 18 Jul 2018 21:42:53 +0300
      Ready:          True
      Restart Count:  0
      Environment:    &lt;none&gt;
      Mounts:
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-mkdlq (ro)
  Containers:
    es-data:
      Container ID:   docker://505269e9be09e83d672b91a582afd7569b3afd794bbfab764d50d75e7a3f7309
      Image:          quay.io/pires/docker-elasticsearch-kubernetes:6.3.0
      Image ID:       docker-pullable://quay.io/pires/docker-elasticsearch-kubernetes@sha256:dcd3e9db3d2c6b9a448d135aebcacac30a4cca655d42efaa115aa57405cd22f3
      Ports:          9200/TCP, 9300/TCP
      Host Ports:     0/TCP, 0/TCP
      State:          Running
        Started:      Wed, 18 Jul 2018 21:46:08 +0300
      Last State:     Terminated
        Reason:       Error
        Exit Code:    143
        Started:      Wed, 18 Jul 2018 21:45:18 +0300
        Finished:     Wed, 18 Jul 2018 21:46:07 +0300
      Ready:          False
      Restart Count:  4
      Limits:
        cpu:  1
      Requests:
        cpu:      250m
      Liveness:   tcp-socket :transport delay=20s timeout=1s period=10s #success=1 #failure=3
      Readiness:  http-get http://:http/_cluster/health delay=20s timeout=5s period=10s #success=1 #failure=3
      Environment:
        NAMESPACE:     default (v1:metadata.namespace)
        NODE_NAME:     es-data-6fdbcf956f-fdnc7 (v1:metadata.name)
        CLUSTER_NAME:  myesdb
        NODE_MASTER:   false
        NODE_INGEST:   false
        HTTP_ENABLE:   true
        ES_JAVA_OPTS:  -Xms256m -Xmx256m
        PROCESSORS:    1 (limits.cpu)
      Mounts:
        /data from storage (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-mkdlq (ro)
    istio-proxy:
      Container ID:  docker://1bf8e167ece0ac6282c336a6630c292013e36721ba027e6e8b5bb71a4bf65a25
      Image:         docker.io/istio/proxyv2:0.8.0
      Image ID:      docker-pullable://istio/proxyv2@sha256:1930f0603321b1917b2249c576ecb4141aaceeaae5fcc0760b6a88dc88daea3e
      Port:          &lt;none&gt;
      Host Port:     &lt;none&gt;
      Args:
        proxy
        sidecar
        --configPath
        /etc/istio/proxy
        --binaryPath
        /usr/local/bin/envoy
        --serviceCluster
        istio-proxy
        --drainDuration
        45s
        --parentShutdownDuration
        1m0s
        --discoveryAddress
        istio-pilot.istio-system:15007
        --discoveryRefreshDelay
        10s
        --zipkinAddress
        zipkin.istio-system:9411
        --connectTimeout
        10s
        --statsdUdpAddress
        istio-statsd-prom-bridge.istio-system:9125
        --proxyAdminPort
        15000
        --controlPlaneAuthPolicy
        NONE
      State:          Running
        Started:      Wed, 18 Jul 2018 21:46:37 +0300
      Last State:     Terminated
        Reason:       Error
        Exit Code:    255
        Started:      Wed, 18 Jul 2018 21:42:55 +0300
        Finished:     Wed, 18 Jul 2018 21:46:36 +0300
      Ready:          True
      Restart Count:  1
      Requests:
        cpu:     100m
        memory:  128Mi
      Environment:
        POD_NAME:                      es-data-6fdbcf956f-fdnc7 (v1:metadata.name)
        POD_NAMESPACE:                 default (v1:metadata.namespace)
        INSTANCE_IP:                    (v1:status.podIP)
        ISTIO_META_POD_NAME:           es-data-6fdbcf956f-fdnc7 (v1:metadata.name)
        ISTIO_META_INTERCEPTION_MODE:  REDIRECT
      Mounts:
        /etc/certs/ from istio-certs (ro)
        /etc/istio/proxy from istio-envoy (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-mkdlq (ro)
  Conditions:
    Type              Status
    Initialized       True 
    Ready             False 
    ContainersReady   False 
    PodScheduled      True 
  Volumes:
    storage:
      Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:  
    istio-envoy:
      Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
      Medium:  Memory
    istio-certs:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  istio.default
      Optional:    true
    default-token-mkdlq:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  default-token-mkdlq
      Optional:    false
  QoS Class:       Burstable
  Node-Selectors:  &lt;none&gt;
  Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                   node.kubernetes.io/unreachable:NoExecute for 300s
  Events:
    Type     Reason     Age               From               Message
    ----     ------     ----              ----               -------
    Normal   Scheduled  5m                default-scheduler  Successfully assigned default/es-data-6fdbcf956f-fdnc7 to node2
    Normal   Pulled     5m                kubelet, node2     Container image "busybox:1.27.2" already present on machine
    Normal   Created    5m                kubelet, node2     Created container
    Normal   Started    5m                kubelet, node2     Started container
    Normal   Started    5m                kubelet, node2     Started container
    Normal   Pulled     5m                kubelet, node2     Container image "docker.io/istio/proxy_init:0.8.0" already present on machine
    Normal   Created    5m                kubelet, node2     Created container
    Normal   Pulled     5m                kubelet, node2     Container image "docker.io/istio/proxyv2:0.8.0" already present on machine
    Normal   Created    5m                kubelet, node2     Created container
    Normal   Started    5m                kubelet, node2     Started container
    Warning  Unhealthy  5m                kubelet, node2     Liveness probe failed: dial tcp 172.16.104.22:9300: connect: invalid argument
    Warning  Unhealthy  5m                kubelet, node2     Readiness probe failed: Get http://172.16.104.22:9200/_cluster/health: dial tcp 172.16.104.22:9200: connect: invalid argument
    Normal   Pulled     5m (x2 over 5m)   kubelet, node2     Container image "quay.io/pires/docker-elasticsearch-kubernetes:6.3.0" already present on machine
    Normal   Created    5m (x2 over 5m)   kubelet, node2     Created container
    Normal   Killing    5m                kubelet, node2     Killing container with id docker://es-data:Container failed liveness probe.. Container will be killed and recreated.
    Normal   Started    5m (x2 over 5m)   kubelet, node2     Started container
    Warning  Unhealthy  4m (x3 over 5m)   kubelet, node2     Readiness probe failed: Get http://172.16.104.22:9200/_cluster/health: dial tcp 172.16.104.22:9200: connect: connection refused
    Warning  Unhealthy  4m (x4 over 5m)   kubelet, node2     Liveness probe failed: dial tcp 172.16.104.22:9300: connect: connection refused
    Warning  Unhealthy  51s (x3 over 1m)  kubelet, node2     (combined from similar events): Readiness probe failed: Get http://172.16.104.22:9200/_cluster/health: EOF
  
  
  
  
  
  
  
  
  root@node1:/home/arkan# k logs pod/es-data-6fdbcf956f-fdnc7 -c es-data
  [2018-07-18T18:46:13,037][INFO ][o.e.n.Node               ] [es-data-6fdbcf956f-fdnc7] initializing ...
  [2018-07-18T18:46:13,267][INFO ][o.e.e.NodeEnvironment    ] [es-data-6fdbcf956f-fdnc7] using [1] data paths, mounts [[/data (/dev/mapper/node1--vg-root)]], net usable_space [13.8gb], net total_space [27.9gb], types [ext4]
  [2018-07-18T18:46:13,270][INFO ][o.e.e.NodeEnvironment    ] [es-data-6fdbcf956f-fdnc7] heap size [247.5mb], compressed ordinary object pointers [true]
  [2018-07-18T18:46:13,272][INFO ][o.e.n.Node               ] [es-data-6fdbcf956f-fdnc7] node name [es-data-6fdbcf956f-fdnc7], node ID [ymKMhUIxRq-hbrmrqzayCQ]
  [2018-07-18T18:46:13,272][INFO ][o.e.n.Node               ] [es-data-6fdbcf956f-fdnc7] version[6.3.0], pid[1], build[default/tar/424e937/2018-06-11T23:38:03.357887Z], OS[Linux/4.4.0-128-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
  [2018-07-18T18:46:13,273][INFO ][o.e.n.Node               ] [es-data-6fdbcf956f-fdnc7] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms256m, -Xmx256m, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
  [2018-07-18T18:46:15,832][WARN ][o.e.d.c.s.Settings       ] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
  [2018-07-18T18:46:18,327][INFO ][o.e.p.PluginsService     ] [es-data-6fdbcf956f-fdnc7] loaded module [aggs-matrix-stats]
  [2018-07-18T18:46:18,331][INFO ][o.e.p.PluginsService     ] [es-data-6fdbcf956f-fdnc7] loaded module [analysis-common]
  [2018-07-18T18:46:18,331][INFO ][o.e.p.PluginsService     ] [es-data-6fdbcf956f-fdnc7] loaded module [ingest-common]
  [2018-07-18T18:46:18,332][INFO ][o.e.p.PluginsService     ] [es-data-6fdbcf956f-fdnc7] loaded module [lang-expression]
  [2018-07-18T18:46:18,332][INFO ][o.e.p.PluginsService     ] [es-data-6fdbcf956f-fdnc7] loaded module [lang-mustache]
  [2018-07-18T18:46:18,337][INFO ][o.e.p.PluginsService     ] [es-data-6fdbcf956f-fdnc7] no plugins loaded
  [2018-07-18T18:46:26,419][INFO ][o.e.x.s.a.s.FileRolesStore] [es-data-6fdbcf956f-fdnc7] parsed [0] roles from file [/elasticsearch/config/roles.yml]
  [2018-07-18T18:46:28,422][INFO ][o.e.d.DiscoveryModule    ] [es-data-6fdbcf956f-fdnc7] using discovery type [zen]
  [2018-07-18T18:46:30,218][INFO ][o.e.n.Node               ] [es-data-6fdbcf956f-fdnc7] initialized
  [2018-07-18T18:46:30,218][INFO ][o.e.n.Node               ] [es-data-6fdbcf956f-fdnc7] starting ...
  [2018-07-18T18:46:30,630][INFO ][o.e.t.TransportService   ] [es-data-6fdbcf956f-fdnc7] publish_address {172.16.104.22:9300}, bound_addresses {172.16.104.22:9300}
  [2018-07-18T18:46:30,701][INFO ][o.e.b.BootstrapChecks    ] [es-data-6fdbcf956f-fdnc7] bound or publishing to a non-loopback address, enforcing bootstrap checks
  [2018-07-18T18:46:33,802][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:46:36,803][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:46:39,805][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
      [2018-07-18T18:46:43,830][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59678, remoteAddress=null}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:46:44,818][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59684, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:46:45,810][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:46:45,902][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59690, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:46:48,812][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:46:48,900][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59714, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:46:49,899][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59720, remoteAddress=null}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:46:51,815][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:46:51,901][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59730, remoteAddress=null}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:46:52,900][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59734, remoteAddress=null}]
   [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:46:59,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59782, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:47:00,781][WARN ][o.e.n.Node               ] [es-data-6fdbcf956f-fdnc7] timed out while waiting for initial discovery state - timeout: 30s
  [2018-07-18T18:47:00,801][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [es-data-6fdbcf956f-fdnc7] publish_address {172.16.104.22:9200}, bound_addresses {172.16.104.22:9200}
  [2018-07-18T18:47:00,803][INFO ][o.e.n.Node               ] [es-data-6fdbcf956f-fdnc7] started
  [2018-07-18T18:47:00,821][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:47:02,896][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59796, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:47:03,822][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:47:06,823][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  
  [2018-07-18T18:47:15,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59880, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:47:18,832][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:47:21,835][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:47:24,837][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:47:24,841][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59936, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:47:27,838][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:47:30,840][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:47:30,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59976, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:47:33,844][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:47:33,898][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:59992, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:47:36,847][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:47:51,854][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:47:51,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:60110, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:47:54,857][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:47:57,858][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  
  [2018-07-18T18:48:06,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:60202, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:48:09,868][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:48:39,886][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:48:39,897][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:60418, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  
  [2018-07-18T18:49:15,909][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:49:15,997][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [es-data-6fdbcf956f-fdnc7] send message failed [channel: NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:60640, remoteAddress=elasticsearch-discovery/172.16.166.167:9300}]
  java.nio.channels.ClosedChannelException: null
   at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2018-07-18T18:49:18,910][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:49:21,912][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:49:24,913][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  [2018-07-18T18:49:27,915][WARN ][o.e.d.z.ZenDiscovery     ] [es-data-6fdbcf956f-fdnc7] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
  </code></pre>