{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/17811","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/17811/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/17811/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/17811/events","html_url":"https://github.com/elastic/elasticsearch/pull/17811","id":148943209,"number":17811,"title":"Persistent Node Ids","user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Cluster","name":":Cluster","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/breaking","name":"breaking","color":"ff0000"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/enhancement","name":"enhancement","color":"4a4ea8"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/review","name":"review","color":"0052cc"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v5.0.0","name":"v5.0.0","color":"dddddd"}],"state":"open","locked":false,"assignee":{"login":"ywelsch","id":3718355,"avatar_url":"https://avatars.githubusercontent.com/u/3718355?v=3","gravatar_id":"","url":"https://api.github.com/users/ywelsch","html_url":"https://github.com/ywelsch","followers_url":"https://api.github.com/users/ywelsch/followers","following_url":"https://api.github.com/users/ywelsch/following{/other_user}","gists_url":"https://api.github.com/users/ywelsch/gists{/gist_id}","starred_url":"https://api.github.com/users/ywelsch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ywelsch/subscriptions","organizations_url":"https://api.github.com/users/ywelsch/orgs","repos_url":"https://api.github.com/users/ywelsch/repos","events_url":"https://api.github.com/users/ywelsch/events{/privacy}","received_events_url":"https://api.github.com/users/ywelsch/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2016-04-17T11:24:59Z","updated_at":"2016-04-19T17:37:50Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/17811","html_url":"https://github.com/elastic/elasticsearch/pull/17811","diff_url":"https://github.com/elastic/elasticsearch/pull/17811.diff","patch_url":"https://github.com/elastic/elasticsearch/pull/17811.patch"},"body":"Node IDs are currently randomly generated during node startup. That means they change every time the node is restarted. While this doesn't matter for ES proper, it makes it hard for external services to track nodes. Another, more minor, side effect is that indexing the output of, say, the node stats API results in creating new fields due to node ID being used as keys.\r\n\r\nThe first approach I considered was to use the node's published address as the base for the id. We already [treat nodes with the same address as the same](https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/discovery/zen/NodeJoinController.java#L387) so this is a simple change (see [here](https://github.com/elastic/elasticsearch/compare/master...bleskes:node_persistent_id_based_on_address)). While this is simple and it works for probably most cases, it is not perfect. For example, if after a node restart, the node is not able to bind to the same port (because it's not yet freed by the OS), it will cause the node to still change identity. Also in environments where the host IP can change due to a host restart, identity will not be the same. \r\n\r\nDue to those limitation, I opted to go with a different approach where the node id will be persisted in the node's data folder. This has the upside of connecting the id to the nodes data. It also means that the host can be adapted in any way (replace network cards, attach storage to a new VM). I\r\n\r\nt does however also have downsides - we now run the risk of two nodes having the same id, if someone copies clones a data folder from one node to another. To mitigate this I changed the semantics of the protection against multiple nodes with the same address to be stricter - it will now reject the incoming join if a node exists with the same id but a different address. Note that if the existing node doesn't respond to pings (i.e., it's not alive) it will be removed and the new node will be accepted when it tries another join.\r\n\r\nLast, and most importantly, this change requires that *all* nodes persist data to disk. This is a change from current behavior where only data & master nodes store local files. This is the main reason for marking this PR as breaking.\r\n\r\nOther less important notes:\r\n- DummyTransportAddress is removed as we need a unique network address per node. Use `LocalTransportAddress.buildUnique()` instead.\r\n- I renamed `node.add_lid_to_custom_path` to `node.add_lock_id_to_custom_path` to avoid confusion with the node ID which is now part of the `NodeEnvironment` logic.\r\n- I removed the `version` paramater from `MetaDataStateFormat#write` , it wasn't really used and was just in the way :)\r\n- TribeNodes are special in the sense that they do start multiple sub-nodes (previously known as client nodes). Those sub-nodes do not store local files and will this have a new id every restart. I have some ideas on how to change this but it will require more sublte changes that I rather be tackled in a follow up issue if needed (/cc @pickypg  @uboness @clintongormley )\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/212034536","html_url":"https://github.com/elastic/elasticsearch/pull/17811#issuecomment-212034536","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/17811","id":212034536,"user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"created_at":"2016-04-19T17:37:50Z","updated_at":"2016-04-19T17:37:50Z","body":"@ywelsch thanks. I addressed your comments and pushed some new commits. Sadly, my Jenkins server signaled an issue with our testing cluster. In the name of reproducibility, it captures the initial seeds of each nodes and when it resets the cluster and start up nodes that were shut down but tests, it will re-use that seed so a node will have the same id. However, because how things work elsewhere, this id may be taken by another node that re-used the data folder of the old node. I added an assertion to LocalDiscovery that surfaces this clearly (had to chase it down, which was sneaky). I'm still thinking about a proper solution. Will ping once it's there."}}