title: filebeat failed to connect to elasticsearch
body: <p>I have <code>Elasticsearch</code> running on Kubernetes (EKS), with <code>filebeat</code> running as <code>daemonset</code> on Kubernetes.</p>
  
  <p>Now I am trying to get the logs from other <code>EC2</code> machines (outside of the EKS), so have installed exact version of <code>filebeat</code> on <code>EC2</code> and configured it to send logs to <code>Elasticsearch</code> running on Kubernetes.</p>
  
  <p>But not able to see any logs in Elasticsearch (Kibana). Here are the logs for filebeat</p>
  
  <pre><code>2019-08-26T18:18:16.005Z    INFO    instance/beat.go:292    Setup Beat: filebeat; Version: 7.2.1
  2019-08-26T18:18:16.005Z    INFO    [index-management]  idxmgmt/std.go:178  Set output.elasticsearch.index to 'filebeat-7.2.1' as ILM is enabled.
  2019-08-26T18:18:16.005Z    INFO    elasticsearch/client.go:166 Elasticsearch url: http://elasticsearch.dev.domain.net:9200
  2019-08-26T18:18:16.005Z    INFO    add_cloud_metadata/add_cloud_metadata.go:351    add_cloud_metadata: hosting provider type detected as aws, metadata={"availability_zone":"us-west-2a","instance":{"id":"i-0185e1d68306f95b4"},"machine":{"type":"t2.medium"},"provider":"aws","region":"us-west-2"}
  2019-08-26T18:18:16.005Z    INFO    [publisher] pipeline/module.go:97   Beat name: dev-web1
  2019-08-26T18:18:16.006Z    INFO    elasticsearch/client.go:166 Elasticsearch url: http://elasticsearch.dev.domain.net:9200
  </code></pre>
  
  <p>Not much info in the logs.</p>
  
  <p>Then I notice :</p>
  
  <pre><code>root@dev-web1:~# sudo systemctl status filebeat
  ● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2019-08-26 18:18:47 UTC; 18min ago
       Docs: https://www.elastic.co/products/beats/filebeat
   Main PID: 7768 (filebeat)
     CGroup: /system.slice/filebeat.service
             └─7768 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs
  
  Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        ERROR        pipeline/output.go:100        Failed to connect to backoff(elasticsearch(http://elasticsear
  Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        INFO        pipeline/output.go:93        Attempting to reconnect to backoff(elasticsearch(http://elastic
  Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        INFO        [publisher]        pipeline/retry.go:189        retryer: send unwait-signal to consumer
  Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:191          done
  Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:166        retryer: send wait signal to consumer
  Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:168          done
  Aug 26 18:35:47 dev-web1 filebeat[7768]: 2019-08-26T18:35:47.028Z        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitori
  Aug 26 18:36:17 dev-web1 filebeat[7768]: 2019-08-26T18:36:17.028Z        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitori
  root@dev-web1:~#
  </code></pre>
  
  <p>But I can't read complete line in above status message. </p>
  
  <p>So I tried :</p>
  
  <pre><code>root@dev-web1:~# curl elasticsearch.dev.domain.net/_cat/health
  1566844775 18:39:35 dev-eks-logs green 3 3 48 24 0 0 0 0 - 100.0%
  root@dev-web1:~#
  </code></pre>
  
  <p>which worked but not with port</p>
  
  <pre><code>root@dev-web1:~# curl elasticsearch.dev.domain.net:9200/_cat/health
  </code></pre>
  
  <p>filebeat has following config</p>
  
  <pre><code>output.elasticsearch:
    hosts: ["elasticsearch.dev.domain.net"]
    username: "elastic"
    password: "changeme"
  </code></pre>
  
  <p>How can I fix this at filebeat side ?</p>
  
  <p>Telnet Test :</p>
  
  <pre><code>root@dev-web1:~# telnet &lt;ip&gt; 5044
  Trying &lt;ip&gt;...
  telnet: Unable to connect to remote host: Connection refused
  root@dev-web1:~# telnet localhost 5044
  Trying 127.0.0.1...
  telnet: Unable to connect to remote host: Connection refused
  root@dev-web1:~#
  </code></pre>