title: Elasticsearch node crashed
body: <p>There are 3 nodes in the cluster running on Ubuntu 20.04, 8 CPU and 32 GB of RAM, 15g allocated to elasticsearch.
  elasticsearch.yml config file:</p>
  <pre><code>---
  discovery.zen.hosts_provider: file
  discovery.zen.minimum_master_nodes: 2
  http.port: &quot;9200-9300&quot;
  network.host: &quot;10.0.6.21&quot;
  node.attr.client: &quot;true&quot;
  node.data: &quot;true&quot;
  node.master: &quot;true&quot;
  node.name: es-server-01
  path.data: /var/lib/elasticsearch/
  path.logs: /var/log/elasticsearch/
  transport.tcp.port: &quot;9300-9400&quot;
  http.cors.enabled: true
  http.cors.allow-origin: &quot;*&quot;
  cluster.name: cluster-prod
  indices.query.bool.max_clause_count: 10000
  </code></pre>
  <p>Log is big, so I'll try to summarize it.</p>
  <p>First, some warnings about G.C.:</p>
  <pre><code>[2022-07-04T14:17:31,094][WARN ][o.e.m.j.JvmGcMonitorService] [es-server-01] [gc][542872] overhead, spent [4.4s] collecting in the last [4.6s]
  [2022-07-04T16:28:23,763][WARN ][o.e.m.j.JvmGcMonitorService] [es-server-01] [gc][550716] overhead, spent [4.8s] collecting in the last [5.1s]
  [2022-07-04T17:11:06,206][WARN ][o.e.m.j.JvmGcMonitorService] [es-server-01] [gc][553272] overhead, spent [4.6s] collecting in the last [5s]
  ...
  [2022-07-04T18:03:35,546][INFO ][o.e.m.j.JvmGcMonitorService] [es-server-01] [gc][old][554244][2459] duration [5s], collections [1]/[5.1s], total [5s]/[45.8m], memory [14.6gb]-&gt;[14.6gb]/[14.9gb], all_pools {[young] [330.4mb]-&gt;[354.6mb]/[532.5mb]}{[survivor] [0b]-&gt;[0b]/[66.5mb]}{[old] [14.3gb]-&gt;[14.3gb]/[14.3gb]}
  [2022-07-04T18:03:35,546][WARN ][o.e.m.j.JvmGcMonitorService] [es-server-01] [gc][554244] overhead, spent [5s] collecting in the last [5.1s]
  ...
  </code></pre>
  <p>Next, complaining about some queue being full?</p>
  <pre><code>[2022-07-04T18:04:17,848][WARN ][r.suppressed             ] [es-server-01] path: /live_api_device/device/_search, params: {index=live_api_device, type=device}
  org.elasticsearch.action.search.SearchPhaseExecutionException: 
          at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:91) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.common.util.concurrent.AbstractRunnable.onRejection(AbstractRunnable.java:63) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.common.util.concurrent.TimedRunnable.onRejection(TimedRunnable.java:50) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onRejection(ThreadContext.java:768) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:104) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.search.AbstractSearchAsyncAction.execute(AbstractSearchAsyncAction.java:311) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.search.FetchSearchPhase.run(FetchSearchPhase.java:80) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:165) ~[elasticsearch-6.8.21.jar:6.8.21]
          ...
          at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.21.jar:6.8.21]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
          at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
  Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@543e8109 on QueueResizingEsThreadPoolExecutor[name = es-server-01/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 86.6ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@57e6901c[Running, pool size = 13, active threads = 13, queued tasks = 1000, completed tasks = 90000591]]
          at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-6.8.21.jar:6.8.21]
          at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_312]
          at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_312]
          at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:98) ~[elasticsearch-6.8.21.jar:6.8.21]
          ... 34 more
  </code></pre>
  <p>Above log goes on for couple of hours, then ends up in OutofMemory</p>
  <pre><code>[2022-07-04T20:49:23,239][WARN ][o.e.t.ThreadPool         ] [es-server-01] failed to run scheduled task [org.elasticsearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker@61b6c8bf] on thread pool [same]
  org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
          at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:680) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:694) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.IndexWriter.getFlushingBytes(IndexWriter.java:578) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.elasticsearch.index.engine.InternalEngine.getWritingBytes(InternalEngine.java:574) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.index.shard.IndexShard.getWritingBytes(IndexShard.java:980) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.indices.IndexingMemoryController.getShardWritingBytes(IndexingMemoryController.java:182) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.runUnlocked(IndexingMemoryController.java:310) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:290) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:247) [elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.21.jar:6.8.21]
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312]
          at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312]
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_312]
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_312]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
          at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
  Caused by: java.lang.OutOfMemoryError: Java heap space
          at org.apache.lucene.util.packed.PackedLongValues$Builder.&lt;init&gt;(PackedLongValues.java:183) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.util.packed.DeltaPackedLongValues$Builder.&lt;init&gt;(DeltaPackedLongValues.java:58) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.util.packed.PackedLongValues.deltaPackedBuilder(PackedLongValues.java:53) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.util.packed.PackedLongValues.deltaPackedBuilder(PackedLongValues.java:58) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.NormValuesWriter.&lt;init&gt;(NormValuesWriter.java:42) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DefaultIndexingChain$PerField.setInvertState(DefaultIndexingChain.java:734) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DefaultIndexingChain$PerField.&lt;init&gt;(DefaultIndexingChain.java:724) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DefaultIndexingChain.getOrAddField(DefaultIndexingChain.java:662) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:428) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1608) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.elasticsearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1308) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:746) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:458) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$3677/1902326889.get(Unknown Source) ~[?:?]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:481) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:456) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:220) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:164) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:156) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:143) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:82) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1059) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1037) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104) ~[elasticsearch-6.8.21.jar:6.8.21]
  [2022-07-04T20:49:23,260][WARN ][o.e.t.OutboundHandler    ] [es-server-01] send message failed [channel: Netty4TcpChannel{localAddress=/10.0.6.21:9300, remoteAddress=/10.0.6.13:41472}]
  java.nio.channels.ClosedChannelException: null
          at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2022-07-04T20:49:28,353][INFO ][o.e.m.j.JvmGcMonitorService] [es-server-01] [gc][557732] overhead, spent [1.2m] collecting in the last [3.9m]
  [2022-07-04T20:49:23,258][WARN ][o.e.t.OutboundHandler    ] [es-server-01] send message failed [channel: Netty4TcpChannel{localAddress=/10.0.6.21:9300, remoteAddress=/10.0.6.13:41470}]
  java.nio.channels.ClosedChannelException: null
          at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2022-07-04T20:49:23,250][WARN ][o.e.t.OutboundHandler    ] [es-server-01] send message failed [channel: Netty4TcpChannel{localAddress=/10.0.6.21:9300, remoteAddress=/10.0.6.13:41474}]
  java.nio.channels.ClosedChannelException: null
          at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2022-07-04T20:49:28,369][WARN ][o.e.t.OutboundHandler    ] [es-server-01] send message failed [channel: Netty4TcpChannel{localAddress=/10.0.6.21:9300, remoteAddress=/10.0.6.13:41470}]
  java.nio.channels.ClosedChannelException: null
          at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2022-07-04T20:49:28,286][WARN ][o.e.t.OutboundHandler    ] [es-server-01] send message failed [channel: Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/10.0.6.13:41450}]
  java.nio.channels.ClosedChannelException: null
          at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2022-07-04T20:49:28,351][WARN ][o.e.t.OutboundHandler    ] [es-server-01] send message failed [channel: Netty4TcpChannel{localAddress=/10.0.6.21:9300, remoteAddress=/10.0.6.13:41472}]
  java.nio.channels.ClosedChannelException: null
          at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  [2022-07-04T20:49:28,347][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-server-01] fatal error in thread [elasticsearch[es-server-01][write][T#1]], exiting
  java.lang.OutOfMemoryError: Java heap space
          at org.apache.lucene.util.packed.PackedLongValues$Builder.&lt;init&gt;(PackedLongValues.java:183) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.util.packed.DeltaPackedLongValues$Builder.&lt;init&gt;(DeltaPackedLongValues.java:58) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.util.packed.PackedLongValues.deltaPackedBuilder(PackedLongValues.java:53) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.util.packed.PackedLongValues.deltaPackedBuilder(PackedLongValues.java:58) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.NormValuesWriter.&lt;init&gt;(NormValuesWriter.java:42) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DefaultIndexingChain$PerField.setInvertState(DefaultIndexingChain.java:734) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DefaultIndexingChain$PerField.&lt;init&gt;(DefaultIndexingChain.java:724) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DefaultIndexingChain.getOrAddField(DefaultIndexingChain.java:662) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:428) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1608) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55]
          at org.elasticsearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1308) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1120) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:826) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:793) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:746) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:458) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$3677/1902326889.get(Unknown Source) ~[?:?]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:481) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:456) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:220) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:164) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:156) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:143) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:82) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1059) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1037) ~[elasticsearch-6.8.21.jar:6.8.21]
          at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104) ~[elasticsearch-6.8.21.jar:6.8.21]
  [2022-07-04T20:49:23,259][WARN ][o.e.t.OutboundHandler    ] [es-server-01] send message failed [channel: Netty4TcpChannel{localAddress=/10.0.6.21:9300, remoteAddress=/10.0.6.13:41464}]
  java.nio.channels.ClosedChannelException: null
          at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source) ~[?:?]
  </code></pre>
  <p>Stats api outputs the following:</p>
  <pre><code>{
    &quot;_nodes&quot; : {
      &quot;total&quot; : 3,
      &quot;successful&quot; : 3,
      &quot;failed&quot; : 0
    },
    &quot;cluster_name&quot; : &quot;cluster-prod&quot;,
    &quot;cluster_uuid&quot; : &quot;D11RC5&lt;redacted&gt;Bw&quot;,
    &quot;timestamp&quot; : 1657015784259,
    &quot;status&quot; : &quot;green&quot;,
    &quot;indices&quot; : {
      &quot;count&quot; : 42,
      &quot;shards&quot; : {
        &quot;total&quot; : 267,
        &quot;primaries&quot; : 142,
        &quot;replication&quot; : 0.8802816901408451,
        &quot;index&quot; : {
          &quot;shards&quot; : {
            &quot;min&quot; : 1,
            &quot;max&quot; : 10,
            &quot;avg&quot; : 6.357142857142857
          },
          &quot;primaries&quot; : {
            &quot;min&quot; : 1,
            &quot;max&quot; : 5,
            &quot;avg&quot; : 3.380952380952381
          },
          &quot;replication&quot; : {
            &quot;min&quot; : 0.0,
            &quot;max&quot; : 1.0,
            &quot;avg&quot; : 0.5952380952380952
          }
        }
      },
      &quot;docs&quot; : {
        &quot;count&quot; : 34969506,
        &quot;deleted&quot; : 652092
      },
      &quot;store&quot; : {
        &quot;size_in_bytes&quot; : 32510407458
      },
      &quot;fielddata&quot; : {
        &quot;memory_size_in_bytes&quot; : 8901884,
        &quot;evictions&quot; : 0
      },
      &quot;query_cache&quot; : {
        &quot;memory_size_in_bytes&quot; : 693628847,
        &quot;total_count&quot; : 703496525,
        &quot;hit_count&quot; : 54393726,
        &quot;miss_count&quot; : 649102799,
        &quot;cache_size&quot; : 332419,
        &quot;cache_count&quot; : 798061,
        &quot;evictions&quot; : 465642
      },
      &quot;completion&quot; : {
        &quot;size_in_bytes&quot; : 884252429
      },
      &quot;segments&quot; : {
        &quot;count&quot; : 1903,
        &quot;memory_in_bytes&quot; : 1087464425,
        &quot;terms_memory_in_bytes&quot; : 1051123719,
        &quot;stored_fields_memory_in_bytes&quot; : 11981304,
        &quot;term_vectors_memory_in_bytes&quot; : 0,
        &quot;norms_memory_in_bytes&quot; : 12152448,
        &quot;points_memory_in_bytes&quot; : 2561246,
        &quot;doc_values_memory_in_bytes&quot; : 9645708,
        &quot;index_writer_memory_in_bytes&quot; : 23989264,
        &quot;version_map_memory_in_bytes&quot; : 3404,
        &quot;fixed_bit_set_memory_in_bytes&quot; : 0,
        &quot;max_unsafe_auto_id_timestamp&quot; : 1643681532685,
        &quot;file_sizes&quot; : { }
      }
    },
    &quot;nodes&quot; : {
      &quot;count&quot; : {
        &quot;total&quot; : 3,
        &quot;data&quot; : 3,
        &quot;coordinating_only&quot; : 0,
        &quot;master&quot; : 3,
        &quot;ingest&quot; : 3
      },
      &quot;versions&quot; : [
        &quot;6.8.21&quot;
      ],
      &quot;os&quot; : {
        &quot;available_processors&quot; : 24,
        &quot;allocated_processors&quot; : 24,
        &quot;names&quot; : [
          {
            &quot;name&quot; : &quot;Linux&quot;,
            &quot;count&quot; : 3
          }
        ],
        &quot;pretty_names&quot; : [
          {
            &quot;pretty_name&quot; : &quot;Ubuntu 20.04.2 LTS&quot;,
            &quot;count&quot; : 3
          }
        ],
        &quot;mem&quot; : {
          &quot;total_in_bytes&quot; : 101031100416,
          &quot;free_in_bytes&quot; : 9266618368,
          &quot;used_in_bytes&quot; : 91764482048,
          &quot;free_percent&quot; : 9,
          &quot;used_percent&quot; : 91
        }
      },
      &quot;process&quot; : {
        &quot;cpu&quot; : {
          &quot;percent&quot; : 21
        },
        &quot;open_file_descriptors&quot; : {
          &quot;min&quot; : 731,
          &quot;max&quot; : 1020,
          &quot;avg&quot; : 917
        }
      },
      &quot;jvm&quot; : {
        &quot;max_uptime_in_millis&quot; : 15101143605,
        &quot;versions&quot; : [
          {
            &quot;version&quot; : &quot;1.8.0_312&quot;,
            &quot;vm_name&quot; : &quot;OpenJDK 64-Bit Server VM&quot;,
            &quot;vm_version&quot; : &quot;25.312-b07&quot;,
            &quot;vm_vendor&quot; : &quot;Private Build&quot;,
            &quot;count&quot; : 3
          }
        ],
        &quot;mem&quot; : {
          &quot;heap_used_in_bytes&quot; : 22552247912,
          &quot;heap_max_in_bytes&quot; : 48109191168
        },
        &quot;threads&quot; : 361
      },
      &quot;fs&quot; : {
        &quot;total_in_bytes&quot; : 404182020096,
        &quot;free_in_bytes&quot; : 304499580928,
        &quot;available_in_bytes&quot; : 283836080128
      },
      &quot;plugins&quot; : [ ],
      &quot;network_types&quot; : {
        &quot;transport_types&quot; : {
          &quot;security4&quot; : 3
        },
        &quot;http_types&quot; : {
          &quot;security4&quot; : 3
        }
      }
    }
  }
  </code></pre>
  <p>Thanks! Any help will be appreciated.</p>