<p dir="auto">Our cluster reached OOM and one node is recovering very slowly after restart.</p>
<p dir="auto">It has a lot of data locally, but doesn't seem to care about it to restore locally:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="web592 ~ # du -ms /var/lib/elasticsearch/statistics/nodes/*
1292882 /var/lib/elasticsearch/statistics/nodes/0"><pre class="notranslate"><code class="notranslate">web592 ~ # du -ms /var/lib/elasticsearch/statistics/nodes/*
1292882 /var/lib/elasticsearch/statistics/nodes/0
</code></pre></div>
<p dir="auto"><code class="notranslate">iostat</code> shows that disks are mostly idle. Cluster is on 1.4.1 with aws plugin for s3 backups. Other 4 nodes recovered their state much more quickly and now cluster is rebalancing.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2014-12-01 19:04:45,186][INFO ][node                     ] [statistics02] version[1.4.1], pid[87174], build[89d3241/2014-11-26T15:49:29Z]
[2014-12-01 19:04:45,186][INFO ][node                     ] [statistics02] initializing ...
[2014-12-01 19:04:45,197][INFO ][plugins                  ] [statistics02] loaded [cloud-aws], sites []
[2014-12-01 19:04:46,729][DEBUG][discovery.zen.elect      ] [statistics02] using minimum_master_nodes [3]
[2014-12-01 19:04:46,730][DEBUG][discovery.zen.ping.multicast] [statistics02] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2014-12-01 19:04:46,733][DEBUG][discovery.zen.ping.unicast] [statistics02] using initial hosts [statistics01.es.pretender.local, statistics02.es.pretender.local, statistics03.es.pretender.local, statistics04.es.pretender.local, statistics05.es.pretender.local], with concurrent_connects [10]
[2014-12-01 19:04:46,743][DEBUG][discovery.zen            ] [statistics02] using ping.timeout [3s], join.timeout [1m], master_election.filter_client [true], master_election.filter_data [false]
[2014-12-01 19:04:46,744][DEBUG][discovery.zen.fd         ] [statistics02] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2014-12-01 19:04:46,746][DEBUG][discovery.zen.fd         ] [statistics02] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2014-12-01 19:04:47,024][DEBUG][gateway.local            ] [statistics02] using initial_shards [quorum], list_timeout [30s]
[2014-12-01 19:04:47,026][DEBUG][indices.recovery         ] [statistics02] using max_bytes_per_sec[200mb], concurrent_streams [5], file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and compress [true]
[2014-12-01 19:04:47,088][DEBUG][gateway.local.state.meta ] [statistics02] using gateway.local.auto_import_dangled [YES], with gateway.local.dangling_timeout [2h]
[2014-12-01 19:04:47,399][DEBUG][gateway.local.state.meta ] [statistics02] took 310ms to load state
[2014-12-01 19:04:49,314][DEBUG][gateway.local.state.shards] [statistics02] took 1.9s to load started shards state
[2014-12-01 19:04:49,324][INFO ][node                     ] [statistics02] initialized
[2014-12-01 19:04:49,324][INFO ][node                     ] [statistics02] starting ...
[2014-12-01 19:04:49,431][INFO ][transport                ] [statistics02] bound_address {inet[/192.168.2.82:9300]}, publish_address {inet[/192.168.2.82:9300]}
[2014-12-01 19:04:49,442][INFO ][discovery                ] [statistics02] statistics/putJb40cQsSRK8HLRaXcBQ
[2014-12-01 19:04:52,459][DEBUG][discovery.zen            ] [statistics02] filtered ping responses: (filter_client[true], filter_data[false])
    --&gt; ping_response{node [[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]]], id[60], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
    --&gt; ping_response{node [[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]]], id[69], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
    --&gt; ping_response{node [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], id[140], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
    --&gt; ping_response{node [[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]]], id[12], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
[2014-12-01 19:05:02,432][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24539
[2014-12-01 19:05:02,434][DEBUG][discovery.zen.fd         ] [statistics02] [master] restarting fault detection against master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], reason [new cluster state received and we are monitoring the wrong master [null]]
[2014-12-01 19:05:02,435][DEBUG][discovery.zen            ] [statistics02] got first state from fresh master [4OG7n-hnTCGUs-81MDxTwQ]
[2014-12-01 19:05:02,435][INFO ][cluster.service          ] [statistics02] detected_master [statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]], added {[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]],[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]],[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]],[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]],}, reason: zen-disco-receive(from master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]])
[2014-12-01 19:05:02,461][INFO ][cluster.routing.allocation.decider] [statistics02] updating [cluster.routing.allocation.disk.watermark.low] to [90%]
[2014-12-01 19:05:02,461][INFO ][cluster.routing.allocation.decider] [statistics02] updating [cluster.routing.allocation.disk.watermark.high] to [95%]
[2014-12-01 19:05:03,298][INFO ][http                     ] [statistics02] bound_address {inet[/192.168.2.82:9200]}, publish_address {inet[/192.168.2.82:9200]}
[2014-12-01 19:05:03,298][INFO ][node                     ] [statistics02] started
[2014-12-01 19:05:12,238][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24540
[2014-12-01 19:05:12,608][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][4] starting recovery from local ...
[2014-12-01 19:05:12,618][DEBUG][index.gateway            ] [statistics02] [statistics-20131231][2] starting recovery from local ...
[2014-12-01 19:05:12,632][DEBUG][index.gateway            ] [statistics02] [statistics-20140505][2] starting recovery from local ...
[2014-12-01 19:05:12,646][DEBUG][index.gateway            ] [statistics02] [statistics-20140726][4] starting recovery from local ...
[2014-12-01 19:05:12,651][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][4] recovery completed from [local], took [43ms]
[2014-12-01 19:05:12,740][DEBUG][index.gateway            ] [statistics02] [statistics-20140505][2] recovery completed from [local], took [108ms]
[2014-12-01 19:05:12,740][DEBUG][index.gateway            ] [statistics02] [statistics-20131231][2] recovery completed from [local], took [122ms]
[2014-12-01 19:05:12,767][DEBUG][index.gateway            ] [statistics02] [statistics-20140726][4] recovery completed from [local], took [121ms]
[2014-12-01 19:05:18,378][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24541
[2014-12-01 19:05:18,421][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][0] starting recovery from local ...
[2014-12-01 19:05:18,424][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][0] recovery completed from [local], took [3ms]
[2014-12-01 19:05:18,430][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][3] starting recovery from local ...
[2014-12-01 19:05:18,433][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][3] recovery completed from [local], took [3ms]
[2014-12-01 19:05:18,441][DEBUG][index.gateway            ] [statistics02] [statistics-20140816][2] starting recovery from local ...
[2014-12-01 19:05:18,505][DEBUG][index.gateway            ] [statistics02] [statistics-20140816][2] recovery completed from [local], took [64ms]
[2014-12-01 19:05:24,440][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24542
[2014-12-01 19:05:24,472][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][2] starting recovery from local ...
[2014-12-01 19:05:24,475][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][2] recovery completed from [local], took [3ms]
[2014-12-01 19:05:24,481][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][1] starting recovery from local ...
[2014-12-01 19:05:24,484][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][1] recovery completed from [local], took [3ms]
[2014-12-01 19:05:24,490][DEBUG][index.gateway            ] [statistics02] [statistics-20140108][2] starting recovery from local ...
[2014-12-01 19:05:24,528][DEBUG][index.gateway            ] [statistics02] [statistics-20140108][2] recovery completed from [local], took [38ms]
[2014-12-01 19:05:28,287][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24543
[2014-12-01 19:05:28,769][DEBUG][indices.recovery         ] [statistics02] [statistics-20140226][1] recovery completed from [[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]]], took [394ms]
[2014-12-01 19:05:32,681][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24544
[2014-12-01 19:05:38,058][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24545
[2014-12-01 19:05:42,821][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24546
[2014-12-01 19:05:52,443][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24547
[2014-12-01 19:06:02,245][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24548
[2014-12-01 19:06:15,962][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24549
[2014-12-01 19:06:23,949][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24550
[2014-12-01 19:06:24,519][DEBUG][indices.recovery         ] [statistics02] [statistics-20140527][4] recovery completed from [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], took [55.6s]
[2014-12-01 19:06:24,520][DEBUG][indices.recovery         ] [statistics02] [statistics-20140413][4] recovery completed from [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], took [45.9s]
[2014-12-01 19:06:28,723][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24551
[2014-12-01 19:06:34,943][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24552
[2014-12-01 19:06:50,748][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24553
[2014-12-01 19:06:58,276][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24554
[2014-12-01 19:06:58,745][DEBUG][indices.recovery         ] [statistics02] [statistics-20140512][3] recovery completed from [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], took [22.7s]
[2014-12-01 19:07:03,330][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24555
[2014-12-01 19:07:08,646][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24556
[2014-12-01 19:07:08,742][DEBUG][indices.recovery         ] [statistics02] [statistics-20140823][3] recovery completed from [[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]]], took [56ms]
[2014-12-01 19:07:13,350][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24557
[2014-12-01 19:07:19,349][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24558
[2014-12-01 19:07:20,403][DEBUG][indices.recovery         ] [statistics02] [statistics-20131121][4] recovery completed from [[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]]], took [7s]
[2014-12-01 19:07:27,139][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24559
[2014-12-01 19:07:27,587][DEBUG][indices.recovery         ] [statistics02] [statistics-20140723][0] recovery completed from [[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]]], took [404ms]
[2014-12-01 19:07:30,786][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24560
[2014-12-01 19:07:35,493][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24561
[2014-12-01 19:07:36,207][DEBUG][indices.recovery         ] [statistics02] [statistics-20140326][1] recovery completed from [[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]]], took [685ms]
[2014-12-01 19:07:39,615][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24562
[2014-12-01 19:07:40,124][DEBUG][indices.recovery         ] [statistics02] [statistics-20140703][2] recovery completed from [[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]]], took [475ms]
[2014-12-01 19:07:44,138][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24563
[2014-12-01 19:07:47,642][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24564
[2014-12-01 19:07:52,212][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24565```"><pre class="notranslate"><code class="notranslate">[2014-12-01 19:04:45,186][INFO ][node                     ] [statistics02] version[1.4.1], pid[87174], build[89d3241/2014-11-26T15:49:29Z]
[2014-12-01 19:04:45,186][INFO ][node                     ] [statistics02] initializing ...
[2014-12-01 19:04:45,197][INFO ][plugins                  ] [statistics02] loaded [cloud-aws], sites []
[2014-12-01 19:04:46,729][DEBUG][discovery.zen.elect      ] [statistics02] using minimum_master_nodes [3]
[2014-12-01 19:04:46,730][DEBUG][discovery.zen.ping.multicast] [statistics02] using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2014-12-01 19:04:46,733][DEBUG][discovery.zen.ping.unicast] [statistics02] using initial hosts [statistics01.es.pretender.local, statistics02.es.pretender.local, statistics03.es.pretender.local, statistics04.es.pretender.local, statistics05.es.pretender.local], with concurrent_connects [10]
[2014-12-01 19:04:46,743][DEBUG][discovery.zen            ] [statistics02] using ping.timeout [3s], join.timeout [1m], master_election.filter_client [true], master_election.filter_data [false]
[2014-12-01 19:04:46,744][DEBUG][discovery.zen.fd         ] [statistics02] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2014-12-01 19:04:46,746][DEBUG][discovery.zen.fd         ] [statistics02] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2014-12-01 19:04:47,024][DEBUG][gateway.local            ] [statistics02] using initial_shards [quorum], list_timeout [30s]
[2014-12-01 19:04:47,026][DEBUG][indices.recovery         ] [statistics02] using max_bytes_per_sec[200mb], concurrent_streams [5], file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and compress [true]
[2014-12-01 19:04:47,088][DEBUG][gateway.local.state.meta ] [statistics02] using gateway.local.auto_import_dangled [YES], with gateway.local.dangling_timeout [2h]
[2014-12-01 19:04:47,399][DEBUG][gateway.local.state.meta ] [statistics02] took 310ms to load state
[2014-12-01 19:04:49,314][DEBUG][gateway.local.state.shards] [statistics02] took 1.9s to load started shards state
[2014-12-01 19:04:49,324][INFO ][node                     ] [statistics02] initialized
[2014-12-01 19:04:49,324][INFO ][node                     ] [statistics02] starting ...
[2014-12-01 19:04:49,431][INFO ][transport                ] [statistics02] bound_address {inet[/192.168.2.82:9300]}, publish_address {inet[/192.168.2.82:9300]}
[2014-12-01 19:04:49,442][INFO ][discovery                ] [statistics02] statistics/putJb40cQsSRK8HLRaXcBQ
[2014-12-01 19:04:52,459][DEBUG][discovery.zen            ] [statistics02] filtered ping responses: (filter_client[true], filter_data[false])
    --&gt; ping_response{node [[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]]], id[60], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
    --&gt; ping_response{node [[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]]], id[69], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
    --&gt; ping_response{node [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], id[140], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
    --&gt; ping_response{node [[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]]], id[12], master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], hasJoinedOnce [true], cluster_name[statistics]}
[2014-12-01 19:05:02,432][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24539
[2014-12-01 19:05:02,434][DEBUG][discovery.zen.fd         ] [statistics02] [master] restarting fault detection against master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], reason [new cluster state received and we are monitoring the wrong master [null]]
[2014-12-01 19:05:02,435][DEBUG][discovery.zen            ] [statistics02] got first state from fresh master [4OG7n-hnTCGUs-81MDxTwQ]
[2014-12-01 19:05:02,435][INFO ][cluster.service          ] [statistics02] detected_master [statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]], added {[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]],[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]],[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]],[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]],}, reason: zen-disco-receive(from master [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]])
[2014-12-01 19:05:02,461][INFO ][cluster.routing.allocation.decider] [statistics02] updating [cluster.routing.allocation.disk.watermark.low] to [90%]
[2014-12-01 19:05:02,461][INFO ][cluster.routing.allocation.decider] [statistics02] updating [cluster.routing.allocation.disk.watermark.high] to [95%]
[2014-12-01 19:05:03,298][INFO ][http                     ] [statistics02] bound_address {inet[/192.168.2.82:9200]}, publish_address {inet[/192.168.2.82:9200]}
[2014-12-01 19:05:03,298][INFO ][node                     ] [statistics02] started
[2014-12-01 19:05:12,238][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24540
[2014-12-01 19:05:12,608][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][4] starting recovery from local ...
[2014-12-01 19:05:12,618][DEBUG][index.gateway            ] [statistics02] [statistics-20131231][2] starting recovery from local ...
[2014-12-01 19:05:12,632][DEBUG][index.gateway            ] [statistics02] [statistics-20140505][2] starting recovery from local ...
[2014-12-01 19:05:12,646][DEBUG][index.gateway            ] [statistics02] [statistics-20140726][4] starting recovery from local ...
[2014-12-01 19:05:12,651][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][4] recovery completed from [local], took [43ms]
[2014-12-01 19:05:12,740][DEBUG][index.gateway            ] [statistics02] [statistics-20140505][2] recovery completed from [local], took [108ms]
[2014-12-01 19:05:12,740][DEBUG][index.gateway            ] [statistics02] [statistics-20131231][2] recovery completed from [local], took [122ms]
[2014-12-01 19:05:12,767][DEBUG][index.gateway            ] [statistics02] [statistics-20140726][4] recovery completed from [local], took [121ms]
[2014-12-01 19:05:18,378][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24541
[2014-12-01 19:05:18,421][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][0] starting recovery from local ...
[2014-12-01 19:05:18,424][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][0] recovery completed from [local], took [3ms]
[2014-12-01 19:05:18,430][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][3] starting recovery from local ...
[2014-12-01 19:05:18,433][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][3] recovery completed from [local], took [3ms]
[2014-12-01 19:05:18,441][DEBUG][index.gateway            ] [statistics02] [statistics-20140816][2] starting recovery from local ...
[2014-12-01 19:05:18,505][DEBUG][index.gateway            ] [statistics02] [statistics-20140816][2] recovery completed from [local], took [64ms]
[2014-12-01 19:05:24,440][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24542
[2014-12-01 19:05:24,472][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][2] starting recovery from local ...
[2014-12-01 19:05:24,475][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][2] recovery completed from [local], took [3ms]
[2014-12-01 19:05:24,481][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][1] starting recovery from local ...
[2014-12-01 19:05:24,484][DEBUG][index.gateway            ] [statistics02] [statistics-20141101-dv][1] recovery completed from [local], took [3ms]
[2014-12-01 19:05:24,490][DEBUG][index.gateway            ] [statistics02] [statistics-20140108][2] starting recovery from local ...
[2014-12-01 19:05:24,528][DEBUG][index.gateway            ] [statistics02] [statistics-20140108][2] recovery completed from [local], took [38ms]
[2014-12-01 19:05:28,287][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24543
[2014-12-01 19:05:28,769][DEBUG][indices.recovery         ] [statistics02] [statistics-20140226][1] recovery completed from [[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]]], took [394ms]
[2014-12-01 19:05:32,681][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24544
[2014-12-01 19:05:38,058][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24545
[2014-12-01 19:05:42,821][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24546
[2014-12-01 19:05:52,443][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24547
[2014-12-01 19:06:02,245][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24548
[2014-12-01 19:06:15,962][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24549
[2014-12-01 19:06:23,949][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24550
[2014-12-01 19:06:24,519][DEBUG][indices.recovery         ] [statistics02] [statistics-20140527][4] recovery completed from [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], took [55.6s]
[2014-12-01 19:06:24,520][DEBUG][indices.recovery         ] [statistics02] [statistics-20140413][4] recovery completed from [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], took [45.9s]
[2014-12-01 19:06:28,723][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24551
[2014-12-01 19:06:34,943][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24552
[2014-12-01 19:06:50,748][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24553
[2014-12-01 19:06:58,276][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24554
[2014-12-01 19:06:58,745][DEBUG][indices.recovery         ] [statistics02] [statistics-20140512][3] recovery completed from [[statistics03][4OG7n-hnTCGUs-81MDxTwQ][web245][inet[/192.168.0.245:9300]]], took [22.7s]
[2014-12-01 19:07:03,330][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24555
[2014-12-01 19:07:08,646][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24556
[2014-12-01 19:07:08,742][DEBUG][indices.recovery         ] [statistics02] [statistics-20140823][3] recovery completed from [[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]]], took [56ms]
[2014-12-01 19:07:13,350][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24557
[2014-12-01 19:07:19,349][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24558
[2014-12-01 19:07:20,403][DEBUG][indices.recovery         ] [statistics02] [statistics-20131121][4] recovery completed from [[statistics01][rgRCHAg_QoiRDpii78xIrw][web190][inet[/192.168.0.190:9300]]], took [7s]
[2014-12-01 19:07:27,139][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24559
[2014-12-01 19:07:27,587][DEBUG][indices.recovery         ] [statistics02] [statistics-20140723][0] recovery completed from [[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]]], took [404ms]
[2014-12-01 19:07:30,786][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24560
[2014-12-01 19:07:35,493][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24561
[2014-12-01 19:07:36,207][DEBUG][indices.recovery         ] [statistics02] [statistics-20140326][1] recovery completed from [[statistics05][S9rUtu4vQtiOknn0vz-04Q][web599][inet[/192.168.2.88:9300]]], took [685ms]
[2014-12-01 19:07:39,615][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24562
[2014-12-01 19:07:40,124][DEBUG][indices.recovery         ] [statistics02] [statistics-20140703][2] recovery completed from [[statistics04][zDkIvouVSfSRkwPRRVyZbw][web467][inet[/192.168.1.212:9300]]], took [475ms]
[2014-12-01 19:07:44,138][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24563
[2014-12-01 19:07:47,642][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24564
[2014-12-01 19:07:52,212][DEBUG][discovery.zen.publish    ] [statistics02] received cluster state version 24565```
</code></pre></div>
<p dir="auto">When restarting a cluster from green state, each shard appears to undergo some form of checksum to verify it before bringing it online.</p>
<p dir="auto">Is there a way to journal writes so that recovery is much much faster, in the way that the xfs filesystem does it.</p>
<p dir="auto">Only review the data that was being written to at the time of the outage or shutdown so that only the in-progress write data needs to be checked.</p>
<p dir="auto">For a clean shutdown, maybe a complete cluster restart command could tell all nodes to shutdown in a clean state then turn off, allowing a near instantaneous recovery on startup. Like stop allocation then flush all translogs, then shut down,etc.</p>
<p dir="auto">Would just take lots of the pain out of cluster restarts.</p>
<p dir="auto">Just an idea</p>