{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/18972","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18972/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18972/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18972/events","html_url":"https://github.com/elastic/elasticsearch/issues/18972","id":161162691,"number":18972,"title":"Corrupt index after disk full, unclean shutdown / Possible to avoid translog recovery at all?","user":{"login":"SergiuCip","id":4825638,"avatar_url":"https://avatars.githubusercontent.com/u/4825638?v=3","gravatar_id":"","url":"https://api.github.com/users/SergiuCip","html_url":"https://github.com/SergiuCip","followers_url":"https://api.github.com/users/SergiuCip/followers","following_url":"https://api.github.com/users/SergiuCip/following{/other_user}","gists_url":"https://api.github.com/users/SergiuCip/gists{/gist_id}","starred_url":"https://api.github.com/users/SergiuCip/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SergiuCip/subscriptions","organizations_url":"https://api.github.com/users/SergiuCip/orgs","repos_url":"https://api.github.com/users/SergiuCip/repos","events_url":"https://api.github.com/users/SergiuCip/events{/privacy}","received_events_url":"https://api.github.com/users/SergiuCip/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Recovery","name":":Recovery","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feedback_needed","name":"feedback_needed","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2016-06-20T10:19:20Z","updated_at":"2016-06-21T12:51:12Z","closed_at":null,"body":"<!--\r\nGitHub is reserved for bug reports and feature requests. The best place\r\nto ask a general question is at the Elastic Discourse forums at\r\nhttps://discuss.elastic.co. If you are in fact posting a bug report or\r\na feature request, please include one and only one of the below blocks\r\nin your new issue.\r\n-->\r\n\r\n<!--\r\nIf you are filing a bug report, please remove the below feature\r\nrequest block and provide responses for all of the below items.\r\n-->\r\n\r\n**Elasticsearch version**: 2.3.1 then after trying the steps below we have decided to update to the newest version, so probably the bugs will have been fixed\r\n\r\n**JVM version**: openjdk version \"1.8.0_91\"\r\nOpenJDK Runtime Environment (build 1.8.0_91-b14)\r\nOpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)\r\n\r\n**OS version**: CentOS 7\r\n\r\n**Description of the problem including expected versus actual behavior**:\r\n\r\nThe disk space on our Elasticsearch server got full and we had to remove some logs in order to reduce the disk space usage. Moreover, we restarted elasticsearch and once we have done so immediately we have seen that some of our indices were broken (please see attached below the logs).\r\n\r\nWe have moved the entire elasticsearch 'data' directory out of the way as it was an urgent to have logs and restarted elasticsearch which started with a fresh 'data' directory.\r\n\r\nIn order to restore the indices we have copied some of them to another Elasticsearch Server because we did not wanted to do this on our Live Elasticsearch server and we did as follows:\r\n\r\n1. Stop the ES server\r\n2. Copy + change permission of the index\r\n3. Restart ES server\r\n\r\nalso:\r\n\r\n- removed all the translog files\r\n\r\nError message:\r\n\r\n[2016-06-20 09:01:56,764][WARN ][cluster.action.shard     ] [Famine] [logstash-2016.05.11][3] received shard failed for target shard [[logstash-2016.05.11][3], node[d-z_xFoqTrGd-kip-Tly_w], [P], v[3], s[INITIALIZING], a[id=yh9ngGZNQ5CIrq87hxZRTg], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-06-20T09:01:56.636Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException; ]]], indexUUID [xN9jy5ZiRqaALYvkDxdFgA], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException; ]\r\n\r\nJun 20 09:01:49 log-01.novalocal elasticsearch[9894]: Caused by: [logstash-2016.05.11][[logstash-2016.05.11][0]] EngineCreationFailureException[fa\r\nJun 20 09:01:49 log-01.novalocal elasticsearch[9894]: at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:155)\r\nJun 20 09:01:49 log-01.novalocal elasticsearch[9894]: at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFac\r\nJun 20 09:01:49 log-01.novalocal elasticsearch[9894]: at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)\r\nJun 20 09:01:49 log-01.novalocal elasticsearch[9894]: at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)\r\nJun 20 09:01:49 log-01.novalocal elasticsearch[9894]: at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:\r\nJun 20 09:01:49 log-01.novalocal elasticsearch[9894]: at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)\r\n\r\n[2016-06-20 07:12:24,538][WARN ][cluster.action.shard     ] [Brain-Child] [logstash-2016.05.11][3] received shard failed for [logstash-2016.05.11][3], node[FcS6VgojTuijenAeemDE6Q], [P], v[3], s[INITIALIZING], a[id=OdCh5n8GSDeZpiCLW-CKZg], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-06-20T07:12:24.364Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException; ]], indexUUID [xN9jy5ZiRqaALYvkDxdFgA], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException; ]\r\n[logstash-2016.05.11][[logstash-2016.05.11][3]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException;\r\n\r\n- removed the translog.ckp\r\n\r\nCaused by: [logstash-2016.05.11][[logstash-2016.05.11][4]] EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2016.05.11/4/translog/translog.ckp];\r\n\tat org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:155)\r\n\tat org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)\r\n\tat org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)\r\n\tat org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)\r\n\tat org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)\r\n\tat org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)\r\n\tat org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)\r\n\r\n- removed everything besides translog.ckp in the translog folder\r\n\r\n- we have also used java -cp lucene-core5.4.1.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex to check our indexes and it said that everything is ok, however still we kept on getting errors.\r\n\r\nCould someone please inform me what I have done wrong and what could I do to restore my indices ?\r\n\r\nThanks,\r\nSergiu"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/227429635","html_url":"https://github.com/elastic/elasticsearch/issues/18972#issuecomment-227429635","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18972","id":227429635,"user":{"login":"SergiuCip","id":4825638,"avatar_url":"https://avatars.githubusercontent.com/u/4825638?v=3","gravatar_id":"","url":"https://api.github.com/users/SergiuCip","html_url":"https://github.com/SergiuCip","followers_url":"https://api.github.com/users/SergiuCip/followers","following_url":"https://api.github.com/users/SergiuCip/following{/other_user}","gists_url":"https://api.github.com/users/SergiuCip/gists{/gist_id}","starred_url":"https://api.github.com/users/SergiuCip/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SergiuCip/subscriptions","organizations_url":"https://api.github.com/users/SergiuCip/orgs","repos_url":"https://api.github.com/users/SergiuCip/repos","events_url":"https://api.github.com/users/SergiuCip/events{/privacy}","received_events_url":"https://api.github.com/users/SergiuCip/received_events","type":"User","site_admin":false},"created_at":"2016-06-21T12:51:11Z","updated_at":"2016-06-21T12:51:11Z","body":"@s1monw You're welcome and thank you for helping me debug it.\r\n\r\n1. No I cannot see the translog-20.ckp file anywhere...I had to create it manually but again it complained.\r\n\r\n2. When the issue happen I was running version 3.2.1\r\n\r\n3. Yes it is correct. When you add the broken index these are the first lines of the log:\r\n\r\n[logstash-2016.05.12][[logstash-2016.05.12][3]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nes\r\nted: NoSuchFileException[/home/scorn/test_es/data/elasticsearch/nodes/0/indices/logstash-2016.05.12/3/translog/translog-21.tlog];\r\n\tat org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)\r\n\tat org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)\r\n\tat org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: [logstash-2016.05.12][[logstash-2016.05.12][3]] EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/home/scorn/test_es/data/elasticse\r\narch/nodes/0/indices/logstash-2016.05.12/3/translog/translog-21.tlog];\r\n\tat org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:155)\r\n\tat org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)\r\n\tat org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)\r\n\tat org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)\r\n\tat org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)\r\n\tat org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)\r\n\tat org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)\r\n\t... 5 more\r\nCaused by: java.nio.file.NoSuchFileException: /home/scorn/test_es/data/elasticsearch/nodes/0/indices/logstash-2016.05.12/3/translog/translog-21.tlog\r\n\tat sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)\r\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)\r\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)\r\n\tat sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)\r\n\tat java.nio.channels.FileChannel.open(FileChannel.java:287)\r\n\tat java.nio.channels.FileChannel.open(FileChannel.java:335)\r\n\tat org.elasticsearch.index.translog.Translog.openReader(Translog.java:374)\r\n\tat org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:334)\r\n\tat org.elasticsearch.index.translog.Translog.<init>(Translog.java:179)\r\n\tat org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:208)\r\n\tat org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:151)\r\n\t... 11 more\r\n[2016-06-21 09:41:30,949][WARN ][cluster.action.shard     ] [Advisor] [logstash-2016.05.12][3] received shard failed for target shard [[logstash-2016.05.12][3], node[Tgnuvv4DSTut\r\ngGCRQpR0ig], [P], v[3], s[INITIALIZING], a[id=ekDqqz2BQEi0ur8Fi7Ow1w], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-06-21T09:41:30.116Z]]], indexUUID [HLREui0SRAO1LZt8ybnT\r\nWA], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: No\r\nSuchFileException[/home/scorn/test_es/data/elasticsearch/nodes/0/indices/logstash-2016.05.12/3/translog/translog-21.tlog]; ]\r\n[logstash-2016.05.12][[logstash-2016.05.12][3]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nes\r\nted: NoSuchFileException[/home/scorn/test_es/data/elasticsearch/nodes/0/indices/logstash-2016.05.12/3/translog/translog-21.tlog];\r\n\tat org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)\r\n\tat org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)\r\n\tat org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: [logstash-2016.05.12][[logstash-2016.05.12][3]] EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/home/scorn/test_es/data/elasticse\r\narch/nodes/0/indices/logstash-2016.05.12/3/translog/translog-21.tlog];\r\n"}}
{"actor":{"display_login":"SergiuCip"}}