<p>I have <code>Elasticsearch</code> running on Kubernetes (EKS), with <code>filebeat</code> running as <code>daemonset</code> on Kubernetes.</p>

<p>Now I am trying to get the logs from other <code>EC2</code> machines (outside of the EKS), so have installed exact version of <code>filebeat</code> on <code>EC2</code> and configured it to send logs to <code>Elasticsearch</code> running on Kubernetes.</p>

<p>But not able to see any logs in Elasticsearch (Kibana). Here are the logs for filebeat</p>

<pre><code>2019-08-26T18:18:16.005Z    INFO    instance/beat.go:292    Setup Beat: filebeat; Version: 7.2.1
2019-08-26T18:18:16.005Z    INFO    [index-management]  idxmgmt/std.go:178  Set output.elasticsearch.index to 'filebeat-7.2.1' as ILM is enabled.
2019-08-26T18:18:16.005Z    INFO    elasticsearch/client.go:166 Elasticsearch url: http://elasticsearch.dev.domain.net:9200
2019-08-26T18:18:16.005Z    INFO    add_cloud_metadata/add_cloud_metadata.go:351    add_cloud_metadata: hosting provider type detected as aws, metadata={"availability_zone":"us-west-2a","instance":{"id":"i-0185e1d68306f95b4"},"machine":{"type":"t2.medium"},"provider":"aws","region":"us-west-2"}
2019-08-26T18:18:16.005Z    INFO    [publisher] pipeline/module.go:97   Beat name: dev-web1
2019-08-26T18:18:16.006Z    INFO    elasticsearch/client.go:166 Elasticsearch url: http://elasticsearch.dev.domain.net:9200
</code></pre>

<p>Not much info in the logs.</p>

<p>Then I notice :</p>

<pre><code>root@dev-web1:~# sudo systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-08-26 18:18:47 UTC; 18min ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 7768 (filebeat)
   CGroup: /system.slice/filebeat.service
           └─7768 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs

Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        ERROR        pipeline/output.go:100        Failed to connect to backoff(elasticsearch(http://elasticsear
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        INFO        pipeline/output.go:93        Attempting to reconnect to backoff(elasticsearch(http://elastic
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.156Z        INFO        [publisher]        pipeline/retry.go:189        retryer: send unwait-signal to consumer
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:191          done
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:166        retryer: send wait signal to consumer
Aug 26 18:35:38 dev-web1 filebeat[7768]: 2019-08-26T18:35:38.157Z        INFO        [publisher]        pipeline/retry.go:168          done
Aug 26 18:35:47 dev-web1 filebeat[7768]: 2019-08-26T18:35:47.028Z        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitori
Aug 26 18:36:17 dev-web1 filebeat[7768]: 2019-08-26T18:36:17.028Z        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitori
root@dev-web1:~#
</code></pre>

<p>But I can't read complete line in above status message. </p>

<p>So I tried :</p>

<pre><code>root@dev-web1:~# curl elasticsearch.dev.domain.net/_cat/health
1566844775 18:39:35 dev-eks-logs green 3 3 48 24 0 0 0 0 - 100.0%
root@dev-web1:~#
</code></pre>

<p>which worked but not with port</p>

<pre><code>root@dev-web1:~# curl elasticsearch.dev.domain.net:9200/_cat/health
</code></pre>

<p>filebeat has following config</p>

<pre><code>output.elasticsearch:
  hosts: ["elasticsearch.dev.domain.net"]
  username: "elastic"
  password: "changeme"
</code></pre>

<p>How can I fix this at filebeat side ?</p>

<p>Telnet Test :</p>

<pre><code>root@dev-web1:~# telnet &lt;ip&gt; 5044
Trying &lt;ip&gt;...
telnet: Unable to connect to remote host: Connection refused
root@dev-web1:~# telnet localhost 5044
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
root@dev-web1:~#
</code></pre>

<p><a href="https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#hosts-option" rel="nofollow noreferrer">https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#hosts-option</a> says:</p>

<blockquote>
  <p><code>hosts</code>...If no port is specified, 9200 is used.</p>
</blockquote>

<p>Adding <code>hosts: ["elasticsearch.dev.domain.net:80"]</code> in the filbeat configuration should resolve the issue.</p>
