<p>I've been running my ECK (Elastic Cloud on Kubernetes) cluster for a couple of weeks with no issues. However, 3 days ago filebeat stopped being able to connect to my ES service. All pods are up and running (Elastic, Beats and Kibana).</p>
<p>Also, shelling into filebeats pods and connecting to the Elasticsearch service works just fine:</p>
<pre class="lang-sh prettyprint-override"><code>curl -k  -u &quot;user:$PASSWORD&quot; https://quickstart-es-http.quickstart.svc:9200
</code></pre>
<pre class="lang-json prettyprint-override"><code>{
  &quot;name&quot; : &quot;aegis-es-default-4&quot;,
  &quot;cluster_name&quot; : &quot;quickstart&quot;,
  &quot;cluster_uuid&quot; : &quot;&quot;,
  &quot;version&quot; : {
    &quot;number&quot; : &quot;7.14.0&quot;,
    &quot;build_flavor&quot; : &quot;default&quot;,
    &quot;build_type&quot; : &quot;docker&quot;,
    &quot;build_hash&quot; : &quot;&quot;,
    &quot;build_date&quot; : &quot;&quot;,
    &quot;build_snapshot&quot; : false,
    &quot;lucene_version&quot; : &quot;8.9.0&quot;,
    &quot;minimum_wire_compatibility_version&quot; : &quot;6.8.0&quot;,
    &quot;minimum_index_compatibility_version&quot; : &quot;6.0.0-beta1&quot;
  },
  &quot;tagline&quot; : &quot;You Know, for Search&quot;
}
</code></pre>
<p>Yet the filebeats pod logs are producing the below error:</p>
<pre><code>ERROR   
[publisher_pipeline_output] pipeline/output.go:154  
Failed to connect to backoff(elasticsearch(https://quickstart-es-http.quickstart.svc:9200)): 
Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: 
503 Service Unavailable: 
{
  &quot;error&quot;: {
    &quot;root_cause&quot;: [
      { &quot;type&quot;: &quot;master_not_discovered_exception&quot;, &quot;reason&quot;: null }
    ],
    &quot;type&quot;: &quot;master_not_discovered_exception&quot;,
    &quot;reason&quot;: null
  },
  &quot;status&quot;: 503
}
</code></pre>
<p>I haven't made any changes so I think it's a case of authentication or SSL certificates needing updating?</p>
<p>My filebeats config looks like this:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
  namespace: quickstart
spec:
  type: filebeat
  version: 7.14.0
  elasticsearchRef:
    name: quickstart
  config:
    filebeat:
      modules:
        - module: gcp
          audit:
            enabled: true
            var.project_id: project_id
            var.topic: topic_name
            var.subcription: sub_name
            var.credentials_file: /usr/certs/credentials_file
            var.keep_original_message: false
          vpcflow:
            enabled: true
            var.project_id: project_id
            var.topic: topic_name
            var.subscription_name: sub_name
            var.credentials_file: /usr/certs/credentials_file
          firewall:
            enabled: true
            var.project_id: project_id
            var.topic: topic_name
            var.subscription_name: sub_name
            var.credentials_file: /usr/certs/credentials_file
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        securityContext:
          runAsUser: 0
        containers:
        - name: filebeat
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
          - name: credentials
            mountPath: /usr/certs
            readOnly: true
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: credentials
          secret:
            defaultMode: 420
            items:
            secretName: elastic-service-account
</code></pre>
<p>And it was working just fine - haven't made any changes to this config to make it lose access.</p>

Lera
<p>Did a little more digging and found that there weren't enough resources to be able to assign a master node.</p>
<p>Got this when I tried to run GET /_cat/master and it returned the same 503 no master error. I added a new node pool and it started running normally.</p>
