<p>I am using <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html" rel="nofollow noreferrer">ECK</a> to deploy Elasticsearch cluster on Kubernetes.</p>
<p>My Elasticsearch is working fine and it shows <code>green</code> as cluster. But when Enterprise search start and start creating <code>indexes</code> in Elasticsearch, after creating some indexes, it give error for timeout.</p>
<p><strong>pv.yaml</strong></p>
<pre><code>---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: elasticsearch-master
  labels:
    type: local
spec:
  storageClassName: standard
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt/nfs/kubernetes/elasticsearch/master/
...
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: elasticsearch-data
  labels:
    type: local
spec:
  storageClassName: standard
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt/nfs/kubernetes/elasticsearch/data/
...
</code></pre>
<p><strong>multi_node.yaml</strong></p>
<pre><code>---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: bselastic
spec:
  version: 8.1.2
  nodeSets:
  - name: masters
    count: 1
    config:
      node.roles: [&quot;master&quot;,
#                  &quot;data&quot;,
                  ]
      xpack.ml.enabled: true
    # Volumeclaim needed to add volume, it was giving error for not volume claim
    # and its not starting pod.
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: standard
  - name: data-node
    count: 1
    config:
      node.roles: [&quot;data&quot;, &quot;ingest&quot;]
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: standard
...
---
apiVersion: enterprisesearch.k8s.elastic.co/v1
kind: EnterpriseSearch
metadata:
  name: enterprise-search-bselastic
spec:
  version: 8.1.3
  count: 1
  elasticsearchRef:
    name: bselastic
  podTemplate:
    spec:
      containers:
      - name: enterprise-search
        env:
        - name: JAVA_OPTS
          value: -Xms2g -Xmx2g
        - name: &quot;elasticsearch.startup_retry.interval&quot;
          value: &quot;30&quot;
        - name: allow_es_settings_modification
          value: &quot;true&quot;
...
</code></pre>
<p>Apply these changes using below command.</p>
<p><code>kubectl apply -f multi_node.yaml -n deleteme -f pv.yaml</code></p>
<p>Check the Elasticsearch cluster status</p>
<pre><code># kubectl get es -n deleteme
NAME        HEALTH    NODES   VERSION   PHASE             AGE
bselastic   unknown           8.1.2     ApplyingChanges   47s
</code></pre>
<p>Check all pods</p>
<pre><code># kubectl get pod -n deleteme
NAME                                               READY   STATUS    RESTARTS   AGE
bselastic-es-data-node-0                           0/1     Running   0          87s
bselastic-es-masters-0                             1/1     Running   0          87s
enterprise-search-bselastic-ent-54675f95f8-9sskf   0/1     Running   0          86s
</code></pre>
<p>Elasticsearch cluster become green after 7+ min</p>
<pre><code>[root@1175014-kubemaster01 nilesh]# kubectl get es -n deleteme
NAME        HEALTH   NODES   VERSION   PHASE   AGE
bselastic   green    2       8.1.2     Ready   7m30s
</code></pre>
<p>enterprise search log</p>
<pre><code># kubectl -n deleteme logs -f enterprise-search-bselastic-ent-549bbcb9-rnhmc
Custom Enterprise Search configuration file detected, not overwriting it (any settings passed via environment will be ignored)
Found java executable in PATH
Java version detected: 11.0.14.1 (major version: 11)
Enterprise Search is starting...
[2022-04-25T16:34:22.282+00:00][7][2000][app-server][INFO]: Elastic Enterprise Search version=8.1.3, JRuby version=9.2.16.0, Ruby version=2.5.7, Rails version=5.2.6
[2022-04-25T16:34:23.862+00:00][7][2000][app-server][INFO]: Performing pre-flight checks for Elasticsearch running on https://bselastic-es-http.deleteme.svc:9200...
[2022-04-25T16:34:25.308+00:00][7][2000][app-server][WARN]: [pre-flight] Failed to connect to Elasticsearch backend. Make sure it is running and healthy.
[2022-04-25T16:34:25.310+00:00][7][2000][app-server][INFO]: [pre-flight] Error: /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:187: Connection refused (Connection refused) (Faraday::ConnectionFailed)

[2022-04-25T16:34:31.353+00:00][7][2000][app-server][WARN]: [pre-flight] Failed to connect to Elasticsearch backend. Make sure it is running and healthy.
[2022-04-25T16:34:31.355+00:00][7][2000][app-server][INFO]: [pre-flight] Error: /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:187: Connection refused (Connection refused) (Faraday::ConnectionFailed)

[2022-04-25T16:34:37.370+00:00][7][2000][app-server][WARN]: [pre-flight] Failed to connect to Elasticsearch backend. Make sure it is running and healthy.
[2022-04-25T16:34:37.372+00:00][7][2000][app-server][INFO]: [pre-flight] Error: /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:187: Connection refused (Connection refused) (Faraday::ConnectionFailed)

[2022-04-25T16:34:43.384+00:00][7][2000][app-server][WARN]: [pre-flight] Failed to connect to Elasticsearch backend. Make sure it is running and healthy.
[2022-04-25T16:34:43.386+00:00][7][2000][app-server][INFO]: [pre-flight] Error: /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:187: Connection refused (Connection refused) (Faraday::ConnectionFailed)

[2022-04-25T16:34:49.400+00:00][7][2000][app-server][WARN]: [pre-flight] Failed to connect to Elasticsearch backend. Make sure it is running and healthy.
[2022-04-25T16:34:49.401+00:00][7][2000][app-server][INFO]: [pre-flight] Error: /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:187: Connection refused (Connection refused) (Faraday::ConnectionFailed)
[2022-04-25T16:37:56.290+00:00][7][2000][app-server][INFO]: [pre-flight] Elasticsearch cluster is ready
[2022-04-25T16:37:56.292+00:00][7][2000][app-server][INFO]: [pre-flight] Successfully connected to Elasticsearch
[2022-04-25T16:37:56.367+00:00][7][2000][app-server][INFO]: [pre-flight] Successfully loaded Elasticsearch plugin information for all nodes
[2022-04-25T16:37:56.381+00:00][7][2000][app-server][INFO]: [pre-flight] Elasticsearch running with an active basic license
[2022-04-25T16:37:56.423+00:00][7][2000][app-server][INFO]: [pre-flight] Elasticsearch API key service is enabled
[2022-04-25T16:37:56.446+00:00][7][2000][app-server][INFO]: [pre-flight] Elasticsearch will be used for authentication
[2022-04-25T16:37:56.447+00:00][7][2000][app-server][INFO]: Elasticsearch looks healthy and configured correctly to run Enterprise Search
[2022-04-25T16:37:56.452+00:00][7][2000][app-server][INFO]: Performing pre-flight checks for Kibana running on http://localhost:5601...
[2022-04-25T16:37:56.482+00:00][7][2000][app-server][WARN]: [pre-flight] Failed to connect to Kibana backend. Make sure it is running and healthy.
[2022-04-25T16:37:56.486+00:00][7][2000][app-server][ERROR]: Could not connect to Kibana backend after 0 seconds.
[2022-04-25T16:37:56.488+00:00][7][2000][app-server][WARN]: Enterprise Search is unable to connect to Kibana. Ensure it is running at http://localhost:5601 for user deleteme-enterprise-search-bselastic-ent-user.
[2022-04-25T16:37:59.344+00:00][7][2000][app-server][INFO]: Elastic APM agent is disabled
{&quot;timestamp&quot;: &quot;2022-04-25T16:38:05+00:00&quot;, &quot;message&quot;: &quot;readiness probe failed&quot;, &quot;curl_rc&quot;: &quot;7&quot;}
{&quot;timestamp&quot;: &quot;2022-04-25T16:38:06+00:00&quot;, &quot;message&quot;: &quot;readiness probe failed&quot;, &quot;curl_rc&quot;: &quot;7&quot;}
{&quot;timestamp&quot;: &quot;2022-04-25T16:38:16+00:00&quot;, &quot;message&quot;: &quot;readiness probe failed&quot;, &quot;curl_rc&quot;: &quot;7&quot;}
{&quot;timestamp&quot;: &quot;2022-04-25T16:38:26+00:00&quot;, &quot;message&quot;: &quot;readiness probe failed&quot;, &quot;curl_rc&quot;: &quot;7&quot;}
{&quot;timestamp&quot;: &quot;2022-04-25T16:38:36+00:00&quot;, &quot;message&quot;: &quot;readiness probe failed&quot;, &quot;curl_rc&quot;: &quot;7&quot;}
[2022-04-25T16:38:43.880+00:00][7][2000][app-server][INFO]: [db_lock] [installation] Status: [Starting] Ensuring migrations tracking index exists
{&quot;timestamp&quot;: &quot;2022-04-25T16:38:45+00:00&quot;, &quot;message&quot;: &quot;readiness probe failed&quot;, &quot;curl_rc&quot;: &quot;7&quot;}
{&quot;timestamp&quot;: &quot;2022-04-25T16:38:56+00:00&quot;, &quot;message&quot;: &quot;readiness probe failed&quot;, &quot;curl_rc&quot;: &quot;7&quot;}
[2022-04-25T16:39:05.283+00:00][7][2000][app-server][INFO]: [db_lock] [installation] Status: [Finished] Ensuring migrations tracking index exists
[2022-04-25T16:39:05.782+00:00][7][2000][app-server][INFO]: [db_lock] [installation] Status: [Starting] Creating indices for 38 models
[2022-05-02T16:21:47.303+00:00][8][2000][es][DEBUG]: {
&quot;request&quot;: {
    &quot;url&quot;: &quot;https://bselastic-es-http.deleteme.svc:9200/.ent-search-actastic-oauth_applications_v2&quot;,
    &quot;method&quot;: &quot;put&quot;,
    &quot;headers&quot;: {
      &quot;Authorization&quot;: &quot;[FILTERED]&quot;,
      &quot;Content-Type&quot;: &quot;application/json&quot;,
      &quot;x-elastic-product-origin&quot;: &quot;enterprise-search&quot;,
      &quot;User-Agent&quot;: &quot;Faraday v1.8.0&quot;
    },
    &quot;params&quot;: null,
    &quot;body&quot;: &quot;{\&quot;settings\&quot;:{\&quot;index\&quot;:{\&quot;hidden\&quot;:true,\&quot;refresh_interval\&quot;:-1},\&quot;number_of_shards\&quot;:1,\&quot;auto_expand_replicas\&quot;:\&quot;0-3\&quot;,\&quot;priority\&quot;:250},\&quot;mappings\&quot;:{\&quot;dynamic\&quot;:\&quot;strict\&quot;,\&quot;properties\&quot;:{\&quot;id\&quot;:{\&quot;type\&quot;:\&quot;keyword\&quot;},\&quot;created_at\&quot;:{\&quot;type\&quot;:\&quot;date\&quot;},\&quot;updated_at\&quot;:{\&quot;type\&quot;:\&quot;date\&quot;},\&quot;name\&quot;:{\&quot;type\&quot;:\&quot;keyword\&quot;},\&quot;uid\&quot;:{\&quot;type\&quot;:\&quot;keyword\&quot;},\&quot;secret\&quot;:{\&quot;type\&quot;:\&quot;keyword\&quot;},\&quot;redirect_uri\&quot;:{\&quot;type\&quot;:\&quot;keyword\&quot;},\&quot;scopes\&quot;:{\&quot;type\&quot;:\&quot;keyword\&quot;},\&quot;confidential\&quot;:{\&quot;type\&quot;:\&quot;boolean\&quot;},\&quot;app_type\&quot;:{\&quot;type\&quot;:\&quot;keyword\&quot;}}},\&quot;aliases\&quot;:{}}&quot;
  },
  &quot;exception&quot;: &quot;/usr/share/enterprise-search/lib/war/lib/swiftype/es/client.class:28: Read timed out (Faraday::TimeoutError)\n&quot;,
  &quot;duration&quot;: 30042.3,
  &quot;stack&quot;: [
    &quot;lib/actastic/schema.class:172:in `create_index!'&quot;,
    &quot;lib/actastic/schema.class:195:in `create_index_and_mapping!'&quot;,
    &quot;shared_togo/lib/shared_togo.class:894:in `block in apply_actastic_migrations'&quot;,
    &quot;shared_togo/lib/shared_togo.class:892:in `block in each'&quot;,
    &quot;shared_togo/lib/shared_togo.class:892:in `block in apply_actastic_migrations'&quot;,
    &quot;lib/db_lock.class:182:in `with_status'&quot;,
    &quot;shared_togo/lib/shared_togo.class:891:in `apply_actastic_migrations'&quot;,
    &quot;shared_togo/lib/shared_togo.class:406:in `block in install!'&quot;,
    &quot;lib/db_lock.class:171:in `with_lock'&quot;,
    &quot;shared_togo/lib/shared_togo.class:399:in `install!'&quot;,
    &quot;config/application.class:102:in `block in Application'&quot;,
    &quot;config/environment.class:9:in `&lt;main&gt;'&quot;,
    &quot;config/environment.rb:1:in `&lt;main&gt;'&quot;,
    &quot;shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'&quot;,
    &quot;shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'&quot;,
    &quot;shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'&quot;,
    &quot;shared_togo/lib/shared_togo/cli.class:125:in `run_command'&quot;,
    &quot;shared_togo/lib/shared_togo/cli.class:112:in `run!'&quot;,
    &quot;bin/enterprise-search-internal:15:in `&lt;main&gt;'&quot;
  ]
}
[2022-04-25T16:55:21.340+00:00][7][2000][app-server][INFO]: [db_lock] [installation] Status: [Failed] Creating indices for 38 models: Error = Faraday::TimeoutError: Read timed out
Unexpected exception while running Enterprise Search:
Error: Read timed out at
</code></pre>
<p>Master node logs</p>
<pre><code># kubectl -n deleteme logs -f bselastic-es-masters-0
Skipping security auto configuration because the configuration file [/usr/share/elasticsearch/config/elasticsearch.yml] is missing or is not a regular file

{&quot;@timestamp&quot;:&quot;2022-04-25T16:55:11.051Z&quot;, &quot;log.level&quot;: &quot;INFO&quot;,  &quot;current.health&quot;:&quot;GREEN&quot;,&quot;message&quot;:&quot;Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ent-search-actastic-search_relevance_suggestions-document_position_id-unique-constraint][0]]]).&quot;,&quot;previous.health&quot;:&quot;YELLOW&quot;,&quot;reason&quot;:&quot;shards started [[.ent-search-actastic-search_relevance_suggestions-document_position_id-unique-constraint][0]]&quot; , &quot;ecs.version&quot;: &quot;1.2.0&quot;,&quot;service.name&quot;:&quot;ES_ECS&quot;,&quot;event.dataset&quot;:&quot;elasticsearch.server&quot;,&quot;process.thread.name&quot;:&quot;elasticsearch[bselastic-es-masters-0][masterService#updateTask][T#1]&quot;,&quot;log.logger&quot;:&quot;org.elasticsearch.cluster.routing.allocation.AllocationService&quot;,&quot;elasticsearch.cluster.uuid&quot;:&quot;rnaZmz4kQwOBNbWau43wYA&quot;,&quot;elasticsearch.node.id&quot;:&quot;YMyOM1umSL22ro86II6Ymw&quot;,&quot;elasticsearch.node.name&quot;:&quot;bselastic-es-masters-0&quot;,&quot;elasticsearch.cluster.name&quot;:&quot;bselastic&quot;}
{&quot;@timestamp&quot;:&quot;2022-04-25T16:55:21.447Z&quot;, &quot;log.level&quot;: &quot;WARN&quot;, &quot;message&quot;:&quot;writing cluster state took [10525ms] which is above the warn threshold of [10s]; [skipped writing] global metadata, wrote metadata for [0] new indices and [1] existing indices, removed metadata for [0] indices and skipped [48] unchanged indices&quot;, &quot;ecs.version&quot;: &quot;1.2.0&quot;,&quot;service.name&quot;:&quot;ES_ECS&quot;,&quot;event.dataset&quot;:&quot;elasticsearch.server&quot;,&quot;process.thread.name&quot;:&quot;elasticsearch[bselastic-es-masters-0][generic][T#5]&quot;,&quot;log.logger&quot;:&quot;org.elasticsearch.gateway.PersistedClusterStateService&quot;,&quot;elasticsearch.cluster.uuid&quot;:&quot;rnaZmz4kQwOBNbWau43wYA&quot;,&quot;elasticsearch.node.id&quot;:&quot;YMyOM1umSL22ro86II6Ymw&quot;,&quot;elasticsearch.node.name&quot;:&quot;bselastic-es-masters-0&quot;,&quot;elasticsearch.cluster.name&quot;:&quot;bselastic&quot;}
{&quot;@timestamp&quot;:&quot;2022-04-25T16:55:21.448Z&quot;, &quot;log.level&quot;: &quot;INFO&quot;, &quot;message&quot;:&quot;after [10.3s] publication of cluster state version [226] is still waiting for {bselastic-es-masters-0}{YMyOM1umSL22ro86II6Ymw}{ljGkLdk-RAukc9NEJtQCVw}{192.168.88.213}{192.168.88.213:9300}{m}{k8s_node_name=1175027-kubeworker15.sb.rackspace.com, xpack.installed=true} [SENT_APPLY_COMMIT], {bselastic-es-data-node-0}{K88khDyfRwaGCBZwMKEaHA}{g9mXrT4WTumoj09W1OylYA}{192.168.88.214}{192.168.88.214:9300}{di}{k8s_node_name=1175027-kubeworker15.sb.rackspace.com, xpack.installed=true} [SENT_PUBLISH_REQUEST]&quot;, &quot;ecs.version&quot;: &quot;1.2.0&quot;,&quot;service.name&quot;:&quot;ES_ECS&quot;,&quot;event.dataset&quot;:&quot;elasticsearch.server&quot;,&quot;process.thread.name&quot;:&quot;elasticsearch[bselastic-es-masters-0][generic][T#1]&quot;,&quot;log.logger&quot;:&quot;org.elasticsearch.cluster.coordination.Coordinator.CoordinatorPublication&quot;,&quot;elasticsearch.cluster.uuid&quot;:&quot;rnaZmz4kQwOBNbWau43wYA&quot;,&quot;elasticsearch.node.id&quot;:&quot;YMyOM1umSL22ro86II6Ymw&quot;,&quot;elasticsearch.node.name&quot;:&quot;bselastic-es-masters-0&quot;,&quot;elasticsearch.cluster.name&quot;:&quot;bselastic&quot;}
</code></pre>
<p>Which attribute we have to set in Enterprise search to increase timeout ? or is there any way to get debug log for Enterprise search ?</p>

Nilesh
<p>You can try to increase the default timeout Globally parameter by following this example:</p>
<pre><code>es = Elasticsearch(timeout=30, max_retries=10, retry_on_timeout=True)
</code></pre>
<p>This would help to give the cluster more time to respond.</p>
