title: Elasticsearch doesn't allow to allocate unassigned shard
body: <p>I have an ES cluster of 2 nodes. As I restarted nodes the cluster status is <code>yellow</code> as some of the shards are <code>unassigned</code>. I've tried to google and the common solution is to reroute <code>unassigned</code> shards. Unfortunately, it doesn't work for me.</p>
  
  <pre><code>curl localhost:9200/_cluster/health?pretty=true
  {
    "cluster_name" : "infra",
    "status" : "yellow",
    "timed_out" : false,
    "number_of_nodes" : 2,
    "number_of_data_nodes" : 2,
    "active_primary_shards" : 34,
    "active_shards" : 68,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 31,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 68.68686868686868
  }
  
  curl localhost:9200/_cluster/settings?pretty
  {
    "persistent" : { },
    "transient" : {
      "cluster" : {
        "routing" : {
          "allocation" : {
            "enable" : "all"
          }
        }
      }
    }
  }
  
  curl localhost:9200/_cat/indices?v
  
  health status index                   pri rep docs.count docs.deleted store.size pri.store.size
  yellow open   logstash-log-2016.05.13   5   2      88314            0    300.5mb        150.2mb
  yellow open   logstash-log-2016.05.12   5   2     254450            0    833.9mb        416.9mb
  yellow open   .kibana                   1   2          3            0     47.8kb         25.2kb
  green  open   .marvel-es-data-1         1   1          3            0      8.7kb          4.3kb
  yellow open   logstash-log-2016.05.11   5   2     313095            0    709.1mb        354.6mb
  yellow open   logstash-log-2016.05.10   5   2     613744            0        1gb        520.2mb
  green  open   .marvel-es-1-2016.05.18   1   1      88720          495     89.9mb           45mb
  green  open   .marvel-es-1-2016.05.17   1   1      69430          492     59.4mb         29.7mb
  yellow open   logstash-log-2016.05.17   5   2     188924            0    518.2mb          259mb
  yellow open   logstash-log-2016.05.18   5   2     226775            0    683.7mb        366.1mb
  </code></pre>
  
  <p>Rerouting</p>
  
  <pre><code>curl -XPOST 'localhost:9200/_cluster/reroute?pretty' -d '{
       "commands": [
          {
              "allocate": {
                  "index": "logstash-log-2016.05.13",
                  "shard": 3,
                  "node": "elasticsearch-mon-1",
                  "allow_primary": true
            }
          }
      ]
    }'
  {
    "error" : {
      "root_cause" : [ {
        "type" : "illegal_argument_exception",
        "reason" : "[allocate] allocation of [logstash-log-2016.05.13][3] on node {elasticsearch-mon-1}{K-J8WKyZRB6bE4031kHkKA}{172.45.0.56}{172.45.0.56:9300} is not allowed, reason: [YES(allocation disabling is ignored)][NO(shard cannot be allocated on same node [K-J8WKyZRB6bE4031kHkKA] it already exists on)][YES(no allocation awareness enabled)][YES(allocation disabling is ignored)][YES(target node version [2.3.2] is same or newer than source node version [2.3.2])][YES(primary is already active)][YES(total shard limit disabled: [index: -1, cluster: -1] &lt;= 0)][YES(shard not primary or relocation disabled)][YES(node passes include/exclude/require filters)][YES(enough disk for shard on node, free: [25.4gb])][YES(below shard recovery limit of [2])]"
      } ],
      "type" : "illegal_argument_exception",
      "reason" : "[allocate] allocation of [logstash-log-2016.05.13][3] on node {elasticsearch-mon-1}{K-J8WKyZRB6bE4031kHkKA}{172.45.0.56}{172.45.0.56:9300} is not allowed, reason: [YES(allocation disabling is ignored)][NO(shard cannot be allocated on same node [K-J8WKyZRB6bE4031kHkKA] it already exists on)][YES(no allocation awareness enabled)][YES(allocation disabling is ignored)][YES(target node version [2.3.2] is same or newer than source node version [2.3.2])][YES(primary is already active)][YES(total shard limit disabled: [index: -1, cluster: -1] &lt;= 0)][YES(shard not primary or relocation disabled)][YES(node passes include/exclude/require filters)][YES(enough disk for shard on node, free: [25.4gb])][YES(below shard recovery limit of [2])]"
    },
    "status" : 400
  }
  
  curl -XPOST 'localhost:9200/_cluster/reroute?pretty' -d '{
       "commands": [
          {
              "allocate": {
                  "index": "logstash-log-2016.05.13",
                  "shard": 3,
                  "node": "elasticsearch-mon-2",
                  "allow_primary": true
            }
          }
      ]
    }'
  {
    "error" : {
      "root_cause" : [ {
        "type" : "illegal_argument_exception",
        "reason" : "[allocate] allocation of [logstash-log-2016.05.13][3] on node {elasticsearch-mon-2}{Rxgq2aWPSVC0pvUW2vBgHA}{172.45.0.166}{172.45.0.166:9300} is not allowed, reason: [YES(allocation disabling is ignored)][NO(shard cannot be allocated on same node [Rxgq2aWPSVC0pvUW2vBgHA] it already exists on)][YES(no allocation awareness enabled)][YES(allocation disabling is ignored)][YES(target node version [2.3.2] is same or newer than source node version [2.3.2])][YES(primary is already active)][YES(total shard limit disabled: [index: -1, cluster: -1] &lt;= 0)][YES(shard not primary or relocation disabled)][YES(node passes include/exclude/require filters)][YES(enough disk for shard on node, free: [25.4gb])][YES(below shard recovery limit of [2])]"
      } ],
      "type" : "illegal_argument_exception",
      "reason" : "[allocate] allocation of [logstash-log-2016.05.13][3] on node {elasticsearch-mon-2}{Rxgq2aWPSVC0pvUW2vBgHA}{172.45.0.166}{172.45.0.166:9300} is not allowed, reason: [YES(allocation disabling is ignored)][NO(shard cannot be allocated on same node [Rxgq2aWPSVC0pvUW2vBgHA] it already exists on)][YES(no allocation awareness enabled)][YES(allocation disabling is ignored)][YES(target node version [2.3.2] is same or newer than source node version [2.3.2])][YES(primary is already active)][YES(total shard limit disabled: [index: -1, cluster: -1] &lt;= 0)][YES(shard not primary or relocation disabled)][YES(node passes include/exclude/require filters)][YES(enough disk for shard on node, free: [25.4gb])][YES(below shard recovery limit of [2])]"
    },
    "status" : 400
  }
  </code></pre>
  
  <p>So it fails and doesn't make any change. Shards are still in unassigned state. </p>
  
  <p>Thank you.</p>
  
  <p>Added</p>
  
  <pre><code>curl localhost:9200/_cat/shards
  
  logstash-log-2016.05.13 2 p STARTED     17706  31.6mb 172.45.0.166 elasticsearch-mon-2
  logstash-log-2016.05.13 2 r STARTED     17706  31.5mb 172.45.0.56  elasticsearch-mon-1
  logstash-log-2016.05.13 2 r UNASSIGNED
  logstash-log-2016.05.13 4 p STARTED     17698  31.6mb 172.45.0.166 elasticsearch-mon-2
  logstash-log-2016.05.13 4 r STARTED     17698  31.4mb 172.45.0.56  elasticsearch-mon-1
  logstash-log-2016.05.13 4 r UNASSIGNED
  </code></pre>