{"action":"closed","issue":{"url":"https://api.github.com/repos/vitessio/vitess/issues/3644","repository_url":"https://api.github.com/repos/vitessio/vitess","labels_url":"https://api.github.com/repos/vitessio/vitess/issues/3644/labels{/name}","comments_url":"https://api.github.com/repos/vitessio/vitess/issues/3644/comments","events_url":"https://api.github.com/repos/vitessio/vitess/issues/3644/events","html_url":"https://github.com/vitessio/vitess/issues/3644","id":295711744,"node_id":"MDU6SXNzdWUyOTU3MTE3NDQ=","number":3644,"title":"Prometheus metrics for Vitess","user":{"login":"zmagg","id":1466845,"node_id":"MDQ6VXNlcjE0NjY4NDU=","avatar_url":"https://avatars1.githubusercontent.com/u/1466845?v=4","gravatar_id":"","url":"https://api.github.com/users/zmagg","html_url":"https://github.com/zmagg","followers_url":"https://api.github.com/users/zmagg/followers","following_url":"https://api.github.com/users/zmagg/following{/other_user}","gists_url":"https://api.github.com/users/zmagg/gists{/gist_id}","starred_url":"https://api.github.com/users/zmagg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zmagg/subscriptions","organizations_url":"https://api.github.com/users/zmagg/orgs","repos_url":"https://api.github.com/users/zmagg/repos","events_url":"https://api.github.com/users/zmagg/events{/privacy}","received_events_url":"https://api.github.com/users/zmagg/received_events","type":"User","site_admin":false},"labels":[{"id":760788586,"node_id":"MDU6TGFiZWw3NjA3ODg1ODY=","url":"https://api.github.com/repos/vitessio/vitess/labels/P3","name":"P3","color":"c2e0c6","default":false},{"id":45388297,"node_id":"MDU6TGFiZWw0NTM4ODI5Nw==","url":"https://api.github.com/repos/vitessio/vitess/labels/Type:%20Feature%20Request","name":"Type: Feature Request","color":"84b6eb","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":19,"created_at":"2018-02-09T00:14:41Z","updated_at":"2018-06-29T17:38:27Z","closed_at":"2018-06-29T17:38:27Z","author_association":"COLLABORATOR","body":"#### Overview\r\nWe’d like to build Prometheus support directly into Vitess, instead of relying on scraping the `/debug/vars`.\r\n\r\n The idea is that we’d publish Prometheus formatted stats at a URL like, http://<host>:<prom_port>/metrics from on each Vitess component. \r\n\r\nThe Prometheus client needs more information than the existing interface provides. It needs:\r\n\r\n-  type of the metric (Counter, Gauge, Histogram, Summary)\r\n-  whether or not the metric is surfaceable in Prom (there are some pure strings in the expvars, like: e.g. `\"BuildTimestamp\": 1514968053` that don’t map to any time series)\r\n- description of the metric\r\n- prefix for what component the metric came from (e.g. ‘vtgate’, ‘vttablet’,...)\r\n\r\nThere are multiple ways the Prometheus client can get this information, some thoughts:\r\n\r\n#### Implementation Idea 1\r\n\r\nPrometheus as built-in, first-class way to export metrics inside Vitess\r\n\r\nIn this implementation, we’d change the existing stats.Publish interface (impacting the other interfaces that call it, like `stats.NewCounters`, `stats.NewTimings`) and all callsites, like so:\r\n\r\n*Old interface:*\r\n\r\n`func Publish(name string, v expvar.Var) `\r\n\r\n*New interface:*\r\n\r\n`func Publish(name string, t promType, desc string, v expvar.Var) `\r\n\r\nIn this implementation, we would change the `stats` package so that when `Publish` was called, we’d also instrument the metric via the Prometheus golang client library.\r\n\r\n#### Implementation Idea 2\r\n\r\nWe could also make Prometheus more plugin-able and optional, instead of built-in. \r\n\r\nIn this implementation, we’d have a Prometheus plugin in every Vitess component that would register as a new available backend that would get called on every expvar callback. The plugin itself would consist of a Vitess-specific wrapper around the Prometheus golang client. On every call to the Prometheus exporter endpoint, the Prometheus registered backend would be triggered and the metrics scraped (just like how the `/debug/vars` expvars work today). \r\n\r\nWe would keep the existing `stats.Publish` interface. Inside the Prometheus plugin, we could write an extensive static mapping function between all of the existing metrics and the information the Prometheus client needs. Whenever somebody adds a new metric, they’d have to add it to this static mapping function explicitly in order for the metric to show up in the Prometheus exported stats. (This looks similar to the existing started-on Prometheus implementation inside Vitess).\r\n\r\n#### Discussion\r\nI favor Idea 1 over Idea 2, mostly for ease of maintenance as Vitess developers add more metrics. It would also make code reading easier for people using Vitess (one place to look for metrics, not two). Also, the addition of the type of metric to the interface would make it easier to integrate other metrics backends (like Dogstatsd/StatsD). \r\n\r\nBut! I think Idea 1 is a more invasive change, and I’m not sure if we want the interface to actually look like that. Thoughts? Tradeoffs? \r\n\r\n"}}
{"actor":{"display_login":"zmagg"}}