{"push_id":2542906690,"size":1,"distinct_size":1,"ref":"refs/heads/gossip-health","head":"0d5cc45b0f05630b1bf7bd4fe393a7b9276d7522","before":"bf10b1812cc03f498cc2317889b2329f743ae098","commits":[{"sha":"0d5cc45b0f05630b1bf7bd4fe393a7b9276d7522","author":{"name":"Tobias Schottdorf","email":"25fad7a39062a81fcaf88abd782f9f69679c0bdf@gmail.com"},"message":"server: add health checks and distress gossip\n\nWhile generating NodeStatus summaries, check for metrics that indicate a\nsevere store, node, or cluster-level problem.\n\nFlagged metrics are gossiped under a newly introduced key. These infos\nare in turn picked up by the newly introduced internal `gossip_alerts`\ntable.\n\nIn effect, operators can monitor the `crdb_internal.gossip_alerts`\ntable on any node (though they'll want to do it on all nodes if there\nare network connectivity issues). Similarly, it'll be straightforward\nto plumb these warnings into the UI, though to the best of my knowledge\nthe UI can't just query `crdb_internal`, and we may expose them in\nanother more suitable location (this is trivial since Gossip is\navailable pretty much everywhere).\n\nFor starters, we only check the metrics for underreplicated and\nunavailable ranges, but there is no limitation on how elaborate these\nhealth checks can become. In fact, they aren't limited to sourcing\nsolely from `NodeStatus`, and in light of #25316 we should consider\nalerting when nodes can't (bidirectionally) communicate.\n\nNB: I had originally envisioned polling the `node_metrics` table\nbecause that allows us to write the health checks in SQL. I had\nthis code but ultimately deleted it as it seemed too roundabout\nand less extensible.\n\nRelease note: None","distinct":true,"url":"https://api.github.com/repos/tschottdorf/cockroach/commits/0d5cc45b0f05630b1bf7bd4fe393a7b9276d7522"}]}
{"actor":{"display_login":"tschottdorf"}}