title: What does `failed to download database [GeoLite2-ASN.mmdb` mean when launching Elasticsearch cluster
body: <p>I deployed an Elasticsearch cluster in AWS EKS with 3 nodes. After launching the cluster, I can see 3 pods are running but 2 of them running fine, one of them keep failing and terminating and restarting.</p>
  <p>Below is the error log on the failed pod.</p>
  <pre><code>{&quot;type&quot;: &quot;server&quot;, &quot;timestamp&quot;: &quot;2021-12-26T08:17:33,061Z&quot;, &quot;level&quot;: &quot;INFO&quot;, &quot;component&quot;: &quot;o.e.i.g.DatabaseRegistry&quot;, &quot;cluster.name&quot;: &quot;elk&quot;, &quot;node.name&quot;: &quot;elk-es-node-1&quot;, &quot;message&quot;: &quot;downloading geoip database [GeoLite2-ASN.mmdb] to [/tmp/elasticsearch-9470345091343635510/geoip-databases/HoGUMQ9ISsCjQ4KhIL2IFA/GeoLite2-ASN.mmdb.tmp.gz]&quot; }
  {&quot;type&quot;: &quot;server&quot;, &quot;timestamp&quot;: &quot;2021-12-26T08:17:33,070Z&quot;, &quot;level&quot;: &quot;ERROR&quot;, &quot;component&quot;: &quot;o.e.i.g.DatabaseRegistry&quot;, &quot;cluster.name&quot;: &quot;elk&quot;, &quot;node.name&quot;: &quot;elk-es-node-1&quot;, &quot;message&quot;: &quot;failed to download database [GeoLite2-ASN.mmdb]&quot;,
  &quot;stacktrace&quot;: [&quot;org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];&quot;,
  &quot;at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:605) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:494) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$3(TransportSearchAction.java:288) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:103) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:76) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:329) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:217) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:93) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:42) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:90) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:70) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:54) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:40) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:390) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:534) ~[elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.ingest.geoip.DatabaseRegistry.lambda$retrieveDatabase$11(DatabaseRegistry.java:359) [ingest-geoip-7.15.2.jar:7.15.2]&quot;,
  &quot;at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.15.2.jar:7.15.2]&quot;,
  &quot;at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]&quot;,
  &quot;at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]&quot;,
  &quot;at java.lang.Thread.run(Thread.java:833) [?:?]&quot;] }
  {&quot;type&quot;: &quot;server&quot;, &quot;timestamp&quot;: &quot;2021-12-26T08:17:33,295Z&quot;, &quot;level&quot;: &quot;INFO&quot;, &quot;component&quot;: &quot;o.e.l.LicenseService&quot;, &quot;cluster.name&quot;: &quot;elk&quot;, &quot;node.name&quot;: &quot;elk-es-node-1&quot;, &quot;message&quot;: &quot;license [8a88ef40-3b0b-439e-9f46-32e911999b7d] mode [basic] - valid&quot; }
  {&quot;type&quot;: &quot;server&quot;, &quot;timestamp&quot;: &quot;2021-12-26T08:17:33,309Z&quot;, &quot;level&quot;: &quot;INFO&quot;, &quot;component&quot;: &quot;o.e.h.AbstractHttpServerTransport&quot;, &quot;cluster.name&quot;: &quot;elk&quot;, &quot;node.name&quot;: &quot;elk-es-node-1&quot;, &quot;message&quot;: &quot;publish_address {elk-es-node-1.elk-es-node.default.svc/10.0.1.182:9200}, bound_addresses {0.0.0.0:9200}&quot;, &quot;cluster.uuid&quot;: &quot;hqRP62pNTze1IWQ0sOOR2Q&quot;, &quot;node.id&quot;: &quot;HoGUMQ9ISsCjQ4KhIL2IFA&quot;  }
  {&quot;type&quot;: &quot;server&quot;, &quot;timestamp&quot;: &quot;2021-12-26T08:17:33,310Z&quot;, &quot;level&quot;: &quot;INFO&quot;, &quot;component&quot;: &quot;o.e.n.Node&quot;, &quot;cluster.name&quot;: &quot;elk&quot;, &quot;node.name&quot;: &quot;elk-es-node-1&quot;, &quot;message&quot;: &quot;started&quot;, &quot;cluster.uuid&quot;: &quot;hqRP62pNTze1IWQ0sOOR2Q&quot;, &quot;node.id&quot;: &quot;HoGUMQ9ISsCjQ4KhIL2IFA&quot;  }
  </code></pre>
  <p>The error message says <code>failed to download database [GeoLite2-ASN.mmdb]</code> but I don't know what does this mean.</p>
  <p>Below is my Elasticsearch K8S spec file.</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: elasticsearch.k8s.elastic.co/v1
  kind: Elasticsearch
  metadata:
    name: elk
  spec:
    version: 7.15.2
    serviceAccountName: docker-sa
    http:
      tls:
        selfSignedCertificate:
          disabled: true
    nodeSets:
    - name: node
      count: 3
      config:
        network.host: 0.0.0.0
        xpack.security.enabled: false
      podTemplate:
        spec:
          initContainers:
          - name: sysctl
            securityContext:
              privileged: true
            command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          containers:
          - name: elasticsearch
            readinessProbe:
              exec:
                command:
                - bash
                - -c
                - /mnt/elastic-internal/scripts/readiness-probe-script.sh
              failureThreshold: 3
              initialDelaySeconds: 10
              periodSeconds: 12
              successThreshold: 1
              timeoutSeconds: 12
            env:
            - name: READINESS_PROBE_TIMEOUT
              value: &quot;120&quot;
            resources:
              requests:
                cpu: 1
                memory: 4Gi
            volumeMounts:
            - name: elasticsearch-data
              mountPath: /usr/share/elasticsearch/data
      volumeClaimTemplates:
      - metadata:
          name: elasticsearch-data
        spec:
          accessModes:
            - ReadWriteOnce
          storageClassName: ebs-sc
          resources:
            requests:
              storage: 1024Gi
  
  </code></pre>
  <p>Any idea why this happens?</p>
  <p>I have provisioned a cluster of nodes (ec2 instance) with 2 desired count and 3 maximum. At the moment, only 2 nodes are launched.</p>
  <p>Below is the output of nodes information:</p>
  <pre><code>$ kubectl get nodes
  NAME                                            STATUS   ROLES    AGE     VERSION
  ip-10-0-1-216.ap-southeast-2.compute.internal   Ready    &lt;none&gt;   6d16h   v1.21.5-eks-bc4871b
  ip-10-0-2-24.ap-southeast-2.compute.internal    Ready    &lt;none&gt;   6d17h   v1.21.5-eks-bc4871b
  
  $ kubectl describe node ip-10-0-1-216.ap-southeast-2.compute.internal
  Name:               ip-10-0-1-216.ap-southeast-2.compute.internal
  Roles:              &lt;none&gt;
  Labels:             beta.kubernetes.io/arch=amd64
                      beta.kubernetes.io/instance-type=t3.xlarge
                      beta.kubernetes.io/os=linux
                      eks.amazonaws.com/capacityType=ON_DEMAND
                      eks.amazonaws.com/nodegroup=elk
                      eks.amazonaws.com/nodegroup-image=ami-045371401c5f70a1e
                      failure-domain.beta.kubernetes.io/region=ap-southeast-2
                      failure-domain.beta.kubernetes.io/zone=ap-southeast-2a
                      kubernetes.io/arch=amd64
                      kubernetes.io/hostname=ip-10-0-1-216.ap-southeast-2.compute.internal
                      kubernetes.io/os=linux
                      node.kubernetes.io/instance-type=t3.xlarge
                      topology.ebs.csi.aws.com/zone=ap-southeast-2a
                      topology.kubernetes.io/region=ap-southeast-2
                      topology.kubernetes.io/zone=ap-southeast-2a
  Annotations:        csi.volume.kubernetes.io/nodeid: {&quot;ebs.csi.aws.com&quot;:&quot;i-0fd067997eddeaf86&quot;}
                      node.alpha.kubernetes.io/ttl: 0
                      volumes.kubernetes.io/controller-managed-attach-detach: true
  CreationTimestamp:  Fri, 24 Dec 2021 20:29:13 +1100
  Taints:             &lt;none&gt;
  Unschedulable:      false
  Lease:
    HolderIdentity:  ip-10-0-1-216.ap-southeast-2.compute.internal
    AcquireTime:     &lt;unset&gt;
    RenewTime:       Fri, 31 Dec 2021 13:25:51 +1100
  Conditions:
    Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
    ----             ------  -----------------                 ------------------                ------                       -------
    MemoryPressure   False   Fri, 31 Dec 2021 13:25:52 +1100   Mon, 27 Dec 2021 20:44:02 +1100   KubeletHasSufficientMemory   kubelet has sufficient memory available
    DiskPressure     False   Fri, 31 Dec 2021 13:25:52 +1100   Fri, 24 Dec 2021 20:29:13 +1100   KubeletHasNoDiskPressure     kubelet has no disk pressure
    PIDPressure      False   Fri, 31 Dec 2021 13:25:52 +1100   Fri, 24 Dec 2021 20:29:13 +1100   KubeletHasSufficientPID      kubelet has sufficient PID available
    Ready            True    Fri, 31 Dec 2021 13:25:52 +1100   Fri, 24 Dec 2021 20:29:34 +1100   KubeletReady                 kubelet is posting ready status
  Addresses:
    InternalIP:   10.0.1.216
    Hostname:     ip-10-0-1-216.ap-southeast-2.compute.internal
    InternalDNS:  ip-10-0-1-216.ap-southeast-2.compute.internal
  Capacity:
    attachable-volumes-aws-ebs:  25
    cpu:                         4
    ephemeral-storage:           20959212Ki
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      16205904Ki
    pods:                        58
  Allocatable:
    attachable-volumes-aws-ebs:  25
    cpu:                         3920m
    ephemeral-storage:           18242267924
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      15189072Ki
    pods:                        58
  System Info:
    Machine ID:                 ec2ac34dd2e6a84bdc340dd6a62c3514
    System UUID:                ec2ac34d-d2e6-a84b-dc34-0dd6a62c3514
    Boot ID:                    26c6dde2-1131-4068-9265-c0c4e12d54d3
    Kernel Version:             5.4.156-83.273.amzn2.x86_64
    OS Image:                   Amazon Linux 2
    Operating System:           linux
    Architecture:               amd64
    Container Runtime Version:  docker://20.10.7
    Kubelet Version:            v1.21.5-eks-bc4871b
    Kube-Proxy Version:         v1.21.5-eks-bc4871b
  ProviderID:                   aws:///ap-southeast-2a/i-0fd067997eddeaf86
  Non-terminated Pods:          (14 in total)
    Namespace                   Name                                            CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
    ---------                   ----                                            ------------  ----------  ---------------  -------------  ---
    cert-manager                cert-manager-68ff46b886-tzfqw                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d16h
    cert-manager                cert-manager-cainjector-7cdbb9c945-w99rf        0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d16h
    cert-manager                cert-manager-webhook-58d45d56b8-wsmwp           0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d16h
    default                     elk-es-node-0                                   1 (25%)       100m (2%)   4Gi (27%)        50Mi (0%)      3d16h
    default                     sidecar-66f887c666-pbbcv                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d19h
    elastic-system              elastic-operator-0                              100m (2%)     1 (25%)     150Mi (1%)       512Mi (3%)     3d16h
    kube-system                 aws-load-balancer-controller-9c59c86d8-jd6j9    100m (2%)     200m (5%)   200Mi (1%)       500Mi (3%)     3d16h
    kube-system                 aws-node-hpnxj                                  10m (0%)      0 (0%)      0 (0%)           0 (0%)         4d14h
    kube-system                 cluster-autoscaler-76fd4db4c-lcff5              100m (2%)     100m (2%)   600Mi (4%)       600Mi (4%)     3d16h
    kube-system                 coredns-68f7974869-x4kxs                        100m (2%)     0 (0%)      70Mi (0%)        170Mi (1%)     3d16h
    kube-system                 ebs-csi-controller-55b9f85d5c-c94m5             0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d16h
    kube-system                 ebs-csi-controller-55b9f85d5c-mt57k             0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d16h
    kube-system                 ebs-csi-node-692n6                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         4d13h
    kube-system                 kube-proxy-cqll5                                100m (2%)     0 (0%)      0 (0%)           0 (0%)         4d14h
  Allocated resources:
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource                    Requests      Limits
    --------                    --------      ------
    cpu                         1510m (38%)   1400m (35%)
    memory                      5116Mi (34%)  1832Mi (12%)
    ephemeral-storage           0 (0%)        0 (0%)
    hugepages-1Gi               0 (0%)        0 (0%)
    hugepages-2Mi               0 (0%)        0 (0%)
    attachable-volumes-aws-ebs  0             0
  Events:                       &lt;none&gt;
  
  Name:               ip-10-0-2-24.ap-southeast-2.compute.internal
  Roles:              &lt;none&gt;
  Labels:             beta.kubernetes.io/arch=amd64
                      beta.kubernetes.io/instance-type=t3.xlarge
                      beta.kubernetes.io/os=linux
                      eks.amazonaws.com/capacityType=ON_DEMAND
                      eks.amazonaws.com/nodegroup=elk
                      eks.amazonaws.com/nodegroup-image=ami-045371401c5f70a1e
                      failure-domain.beta.kubernetes.io/region=ap-southeast-2
                      failure-domain.beta.kubernetes.io/zone=ap-southeast-2b
                      kubernetes.io/arch=amd64
                      kubernetes.io/hostname=ip-10-0-2-24.ap-southeast-2.compute.internal
                      kubernetes.io/os=linux
                      node.kubernetes.io/instance-type=t3.xlarge
                      topology.ebs.csi.aws.com/zone=ap-southeast-2b
                      topology.kubernetes.io/region=ap-southeast-2
                      topology.kubernetes.io/zone=ap-southeast-2b
  Annotations:        csi.volume.kubernetes.io/nodeid: {&quot;ebs.csi.aws.com&quot;:&quot;i-0bac62ee2ae10a59a&quot;}
                      node.alpha.kubernetes.io/ttl: 0
                      volumes.kubernetes.io/controller-managed-attach-detach: true
  CreationTimestamp:  Fri, 24 Dec 2021 20:23:12 +1100
  Taints:             &lt;none&gt;
  Unschedulable:      false
  Lease:
    HolderIdentity:  ip-10-0-2-24.ap-southeast-2.compute.internal
    AcquireTime:     &lt;unset&gt;
    RenewTime:       Fri, 31 Dec 2021 13:26:12 +1100
  Conditions:
    Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
    ----             ------  -----------------                 ------------------                ------                       -------
    MemoryPressure   False   Fri, 31 Dec 2021 13:25:52 +1100   Mon, 27 Dec 2021 20:23:13 +1100   KubeletHasSufficientMemory   kubelet has sufficient memory available
    DiskPressure     False   Fri, 31 Dec 2021 13:25:52 +1100   Fri, 24 Dec 2021 20:23:12 +1100   KubeletHasNoDiskPressure     kubelet has no disk pressure
    PIDPressure      False   Fri, 31 Dec 2021 13:25:52 +1100   Fri, 24 Dec 2021 20:23:12 +1100   KubeletHasSufficientPID      kubelet has sufficient PID available
    Ready            True    Fri, 31 Dec 2021 13:25:52 +1100   Fri, 24 Dec 2021 20:23:32 +1100   KubeletReady                 kubelet is posting ready status
  Addresses:
    InternalIP:   10.0.2.24
    Hostname:     ip-10-0-2-24.ap-southeast-2.compute.internal
    InternalDNS:  ip-10-0-2-24.ap-southeast-2.compute.internal
  Capacity:
    attachable-volumes-aws-ebs:  25
    cpu:                         4
    ephemeral-storage:           20959212Ki
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      16205904Ki
    pods:                        58
  Allocatable:
    attachable-volumes-aws-ebs:  25
    cpu:                         3920m
    ephemeral-storage:           18242267924
    hugepages-1Gi:               0
    hugepages-2Mi:               0
    memory:                      15189072Ki
    pods:                        58
  System Info:
    Machine ID:                 ec29ef8b81ba9ab1229c8348fa8f6c97
    System UUID:                ec29ef8b-81ba-9ab1-229c-8348fa8f6c97
    Boot ID:                    8bf36296-625d-4749-8e81-abe0d7dd85c8
    Kernel Version:             5.4.156-83.273.amzn2.x86_64
    OS Image:                   Amazon Linux 2
    Operating System:           linux
    Architecture:               amd64
    Container Runtime Version:  docker://20.10.7
    Kubelet Version:            v1.21.5-eks-bc4871b
    Kube-Proxy Version:         v1.21.5-eks-bc4871b
  ProviderID:                   aws:///ap-southeast-2b/i-0bac62ee2ae10a59a
  Non-terminated Pods:          (7 in total)
    Namespace                   Name                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
    ---------                   ----                          ------------  ----------  ---------------  -------------  ---
    default                     elk-es-node-1                 1 (25%)       100m (2%)   4Gi (27%)        50Mi (0%)      3d16h
    default                     kibana-kb-7f66d6978d-6knrx    100m (2%)     100m (2%)   1Gi (6%)         1Gi (6%)       3d16h
    default                     transform-798f5758cd-4x94w    1 (25%)       0 (0%)      2Gi (13%)        0 (0%)         6d16h
    kube-system                 aws-node-qmfg7                10m (0%)      0 (0%)      0 (0%)           0 (0%)         6d17h
    kube-system                 coredns-68f7974869-wqs2k      100m (2%)     0 (0%)      70Mi (0%)        170Mi (1%)     3d16h
    kube-system                 ebs-csi-node-48h2n            0 (0%)        0 (0%)      0 (0%)           0 (0%)         6d17h
    kube-system                 kube-proxy-f99dh              100m (2%)     0 (0%)      0 (0%)           0 (0%)         6d17h
  Allocated resources:
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource                    Requests      Limits
    --------                    --------      ------
    cpu                         2310m (58%)   200m (5%)
    memory                      7238Mi (48%)  1244Mi (8%)
    ephemeral-storage           0 (0%)        0 (0%)
    hugepages-1Gi               0 (0%)        0 (0%)
    hugepages-2Mi               0 (0%)        0 (0%)
    attachable-volumes-aws-ebs  0             0
  Events:                       &lt;none&gt;
  
  </code></pre>