{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/4309","repository_url":"https://api.github.com/repos/prometheus/prometheus","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/4309/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/4309/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/4309/events","html_url":"https://github.com/prometheus/prometheus/issues/4309","id":335420335,"node_id":"MDU6SXNzdWUzMzU0MjAzMzU=","number":4309,"title":"Consistent hashing in addition to hashmod","user":{"login":"brancz","id":4546722,"node_id":"MDQ6VXNlcjQ1NDY3MjI=","avatar_url":"https://avatars3.githubusercontent.com/u/4546722?v=4","gravatar_id":"","url":"https://api.github.com/users/brancz","html_url":"https://github.com/brancz","followers_url":"https://api.github.com/users/brancz/followers","following_url":"https://api.github.com/users/brancz/following{/other_user}","gists_url":"https://api.github.com/users/brancz/gists{/gist_id}","starred_url":"https://api.github.com/users/brancz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brancz/subscriptions","organizations_url":"https://api.github.com/users/brancz/orgs","repos_url":"https://api.github.com/users/brancz/repos","events_url":"https://api.github.com/users/brancz/events{/privacy}","received_events_url":"https://api.github.com/users/brancz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2018-06-25T14:00:25Z","updated_at":"2018-07-02T16:08:31Z","closed_at":null,"author_association":"MEMBER","body":"The `hashmod` relabeling action is already super useful in order to shard Prometheus targets onto multiple Prometheus instances, without having to treat Prometheus as a distributed system. However, `hashmod` uses a [md5 hash](https://github.com/prometheus/prometheus/blob/2bd510a63e48ac6bf4971d62199bdb1045c93f1a/relabel/relabel.go#L73-L75) which is evenly distributed, but in contrast to consistent hashing methods on the event of changing the number of shards, the distribution of targets to nodes is very disruptive.\r\n\r\nWith this issue I propose to add an additional relabeling action, that implements a consistent hashing method that can be used in the same way as `hashmod`, but less disruptive in terms of re-sharding of targets when changing the number of shards.\r\n\r\nI would also like to use this issue to start the discussion of which consistent hashing algorithm we should choose to accomplish this. I have not performed any experiments on this yet, so any experience and knowledge would be highly appreciated.\r\n\r\nIn terms of characteristics/trade-offs of the system we don't care much about lookup speed, as relabeling results are cached for target-config combinations, so we can focus on getting the best distribution and least disruptive re-sharding. I have no experience with this, but I feel we could even do some consistent hashing with bounded loads using the scrape limits as the \"bounded load\" (only I'm not sure how to describe maximum load other than having it be a new configuration field, but I guess that's what we have the discussion comments here for :slightly_smiling_face:).\r\n\r\n@brian-brazil @tomwilkie I remember we talked about this once or twice at KubeCon Copenhagen, let me know if you have any other insights.\r\n\r\ncc @mxinden @Bplotka @domgreen "},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/401855004","html_url":"https://github.com/prometheus/prometheus/issues/4309#issuecomment-401855004","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/4309","id":401855004,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTg1NTAwNA==","user":{"login":"brancz","id":4546722,"node_id":"MDQ6VXNlcjQ1NDY3MjI=","avatar_url":"https://avatars3.githubusercontent.com/u/4546722?v=4","gravatar_id":"","url":"https://api.github.com/users/brancz","html_url":"https://github.com/brancz","followers_url":"https://api.github.com/users/brancz/followers","following_url":"https://api.github.com/users/brancz/following{/other_user}","gists_url":"https://api.github.com/users/brancz/gists{/gist_id}","starred_url":"https://api.github.com/users/brancz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brancz/subscriptions","organizations_url":"https://api.github.com/users/brancz/orgs","repos_url":"https://api.github.com/users/brancz/repos","events_url":"https://api.github.com/users/brancz/events{/privacy}","received_events_url":"https://api.github.com/users/brancz/received_events","type":"User","site_admin":false},"created_at":"2018-07-02T16:08:31Z","updated_at":"2018-07-02T16:08:31Z","author_association":"MEMBER","body":"Yeah I wrote a PoC where a prometheus would drop all targets that were not “scheduled onto it” (meaning any Kubernetes pod that had a specific annotation) and then had an additional scheduler, das would add the annotation to a pod. While this works and scheduling can be done quite dynamically and without any additional data store (in the Kubernetes case) it's very difficult to determine whether this system is actually working. This is as Brian mentioned a full on distributed monitoring system."}}
{"actor":{"display_login":"brancz"}}