{"action":"opened","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/18289","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18289/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18289/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18289/events","html_url":"https://github.com/elastic/elasticsearch/issues/18289","id":154401225,"number":18289,"title":"failed to upgrade from 1.7.4 to 2.3.2, initializing primary shards stuck in TRANSLOG stage.","user":{"login":"sweetest","id":6868049,"avatar_url":"https://avatars.githubusercontent.com/u/6868049?v=3","gravatar_id":"","url":"https://api.github.com/users/sweetest","html_url":"https://github.com/sweetest","followers_url":"https://api.github.com/users/sweetest/followers","following_url":"https://api.github.com/users/sweetest/following{/other_user}","gists_url":"https://api.github.com/users/sweetest/gists{/gist_id}","starred_url":"https://api.github.com/users/sweetest/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sweetest/subscriptions","organizations_url":"https://api.github.com/users/sweetest/orgs","repos_url":"https://api.github.com/users/sweetest/repos","events_url":"https://api.github.com/users/sweetest/events{/privacy}","received_events_url":"https://api.github.com/users/sweetest/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-05-12T05:37:52Z","updated_at":"2016-05-12T05:37:52Z","closed_at":null,"body":"when I restarted cluster, some of the primaries are properly restored, but some recovery task tasks hang\r\n\r\nthey are in the 'TRANSLOG' stage and are staying there for more than an hour, which is not expected since I run _flush before restart meaning there should be no remaining translog.\r\n\r\nhot threads api shows below\r\n\r\n```\r\n0.0% (97.7micros out of 500ms) cpu usage by thread 'elasticsearch[IP][transport_client_timer][T#1]{Hashed wheel timer #1}'\r\n 10/10 snapshots sharing following 5 elements\r\n   java.lang.Thread.sleep(Native Method)\r\n   org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)\r\n   org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)\r\n   org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)\r\n```\r\n\r\nwhile /_cat/recovery shows a lot of lines just like below\r\n\r\n```\r\nINDEX_NAME 9 1688129 store translog 10.96.250.224 10.96.250.224 n/a n/a 0 100.0% 0 100.0% 111 5087612156 0 -1.0% -1\r\n```\r\n\r\nsome of the relevant error logs in data nodes are as below.\r\n\r\n```\r\n[2016-05-12 12:00:54,273][WARN ][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop.\r\njava.lang.OutOfMemoryError: Java heap space\r\n```\r\n\r\n```\r\n2016-05-12 12:00:54,940][DEBUG][action.admin.indices.stats] [xseed036.kdev] [indices:monitor/stats] failed to execute operation for shard [[v2-nelo2-log-2016-04-13][3], node[kji2YtUgQ_O6MCh40Tb26A], [P], v[15], s[INITIALIZING], a[id=ZggyMnC7Sfm3ZHhYv2YxUg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-05-12T02:59:33.040Z]]]\r\n[v2-nelo2-log-2016-04-13][[v2-nelo2-log-2016-04-13][3]] BroadcastShardOperationFailedException[operation indices:monitor/stats failed]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];\r\n        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:399)\r\n        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:376)\r\n        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:365)\r\n        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:299)\r\n        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: [v2-nelo2-log-2016-04-13][[v2-nelo2-log-2016-04-13][3]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]\r\n        at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:957)\r\n        at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:791)\r\n        at org.elasticsearch.index.shard.IndexShard.docStats(IndexShard.java:612)\r\n        at org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:131)\r\n        at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:165)\r\n        at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:47)\r\n        at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:395)\r\n        ... 7 more\r\n```\r\n\r\nI tried again to upgrade from 1.7.4 to 2.0.0, just out of curiosity, but similar problem occurred. Do you have any idea on why this is happening?"}}