{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/16763","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16763/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16763/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16763/events","html_url":"https://github.com/elastic/elasticsearch/issues/16763","id":135359929,"number":16763,"title":"very uneven distribution of empty shards on multiple data paths if the free spaces are different","user":{"login":"Dieken","id":2365,"avatar_url":"https://avatars.githubusercontent.com/u/2365?v=3","gravatar_id":"","url":"https://api.github.com/users/Dieken","html_url":"https://github.com/Dieken","followers_url":"https://api.github.com/users/Dieken/followers","following_url":"https://api.github.com/users/Dieken/following{/other_user}","gists_url":"https://api.github.com/users/Dieken/gists{/gist_id}","starred_url":"https://api.github.com/users/Dieken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dieken/subscriptions","organizations_url":"https://api.github.com/users/Dieken/orgs","repos_url":"https://api.github.com/users/Dieken/repos","events_url":"https://api.github.com/users/Dieken/events{/privacy}","received_events_url":"https://api.github.com/users/Dieken/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-02-22T09:17:39Z","updated_at":"2016-02-22T16:54:03Z","closed_at":null,"body":"I met this issue on ES-2.2.0,  all ES data nodes have multiple hard disks and no RAID scheme is utilized, they are just a bunch of disks. The usable spaces of those disks on same host are different due to some strange implementation details in XFS (even two XFS fs have same capacity, same size directories and files, they don't always have same free space, I guess it's produced by file creations and deletions).\r\n\r\nI pre-create many indicies with replica set to 0,  50 shards for 50 data nodes. The initial empty shard is very small, just several KB disk usage,  ES surprisingly creates a lot of shards on the least used disks,  for example, if one of ten disks on a data node has 10KB more free space,  then it will contain about 8 more shards than other disks.  The least used disks will be run out because I use hourly index pattern,  all shards for a continuous hour range will go to the same disk.  But other disks on that host still have a lot of free space.\r\n\r\nI checked a little the code, https://github.com/elastic/elasticsearch/blob/v2.2.0/core/src/main/java/org/elasticsearch/index/IndexService.java#L327, the IndexService instance is for per index, not a singleton, so \"dataPathToShardCount\" will always be empty due to zero replica for this index, so at https://github.com/elastic/elasticsearch/blob/v2.2.0/core/src/main/java/org/elasticsearch/index/shard/ShardPath.java#L231,  \"count\" will always be null,  class \"ShardPath\" always selects the least used disk to allocate shard,  but unluckily the empty shard is very small,  so a lot of shards will be allocated to the least used disk.\r\n\r\nThis issue isn't limited to zero replica use case,  the current ES logic will always prefer the least used disk and the very small empty shard always produces uneven distribution.\r\n\r\nI feel the \"dataPathToShardCount\" map should be global to all indices on a node, not local to a single index,  but maybe there is better solution.\r\n\r\nMy workaround is to create RAID 0 for those disks on a single node, but RAID 0 is too risky if a host has about 10 disks, maybe I should choose RAID 6."},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/187263205","html_url":"https://github.com/elastic/elasticsearch/issues/16763#issuecomment-187263205","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/16763","id":187263205,"user":{"login":"dakrone","id":19060,"avatar_url":"https://avatars.githubusercontent.com/u/19060?v=3","gravatar_id":"","url":"https://api.github.com/users/dakrone","html_url":"https://github.com/dakrone","followers_url":"https://api.github.com/users/dakrone/followers","following_url":"https://api.github.com/users/dakrone/following{/other_user}","gists_url":"https://api.github.com/users/dakrone/gists{/gist_id}","starred_url":"https://api.github.com/users/dakrone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dakrone/subscriptions","organizations_url":"https://api.github.com/users/dakrone/orgs","repos_url":"https://api.github.com/users/dakrone/repos","events_url":"https://api.github.com/users/dakrone/events{/privacy}","received_events_url":"https://api.github.com/users/dakrone/received_events","type":"User","site_admin":false},"created_at":"2016-02-22T16:54:03Z","updated_at":"2016-02-22T16:54:03Z","body":"https://github.com/elastic/elasticsearch/pull/12947 may also be at work here too"}}