{"action":"created","issue":{"url":"https://api.github.com/repos/divolte/divolte-collector/issues/195","repository_url":"https://api.github.com/repos/divolte/divolte-collector","labels_url":"https://api.github.com/repos/divolte/divolte-collector/issues/195/labels{/name}","comments_url":"https://api.github.com/repos/divolte/divolte-collector/issues/195/comments","events_url":"https://api.github.com/repos/divolte/divolte-collector/issues/195/events","html_url":"https://github.com/divolte/divolte-collector/issues/195","id":304693078,"number":195,"title":"prometheus metrics","user":{"login":"leo-baltus","id":8583808,"avatar_url":"https://avatars3.githubusercontent.com/u/8583808?v=4","gravatar_id":"","url":"https://api.github.com/users/leo-baltus","html_url":"https://github.com/leo-baltus","followers_url":"https://api.github.com/users/leo-baltus/followers","following_url":"https://api.github.com/users/leo-baltus/following{/other_user}","gists_url":"https://api.github.com/users/leo-baltus/gists{/gist_id}","starred_url":"https://api.github.com/users/leo-baltus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leo-baltus/subscriptions","organizations_url":"https://api.github.com/users/leo-baltus/orgs","repos_url":"https://api.github.com/users/leo-baltus/repos","events_url":"https://api.github.com/users/leo-baltus/events{/privacy}","received_events_url":"https://api.github.com/users/leo-baltus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-03-13T09:49:31Z","updated_at":"2018-04-20T13:51:28Z","closed_at":null,"author_association":"NONE","body":"We had an issue where one of the sinks was not properly configured which resulted in divolte trying non-stop deliver to it. Meanwhile the other sink was working and divolte was happily taking in traffic so we did not notice right away.\r\n\r\nOne way of detecting this situation is to have divolte report its health (in a more granular manner than a ping) through metrics. Since we are using prometheus and kubernetes it would be nice to have some metrics as well as an idea of how happy divolte is at any given time.\r\n"},"comment":{"url":"https://api.github.com/repos/divolte/divolte-collector/issues/comments/383102580","html_url":"https://github.com/divolte/divolte-collector/issues/195#issuecomment-383102580","issue_url":"https://api.github.com/repos/divolte/divolte-collector/issues/195","id":383102580,"user":{"login":"friso","id":273638,"avatar_url":"https://avatars1.githubusercontent.com/u/273638?v=4","gravatar_id":"","url":"https://api.github.com/users/friso","html_url":"https://github.com/friso","followers_url":"https://api.github.com/users/friso/followers","following_url":"https://api.github.com/users/friso/following{/other_user}","gists_url":"https://api.github.com/users/friso/gists{/gist_id}","starred_url":"https://api.github.com/users/friso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/friso/subscriptions","organizations_url":"https://api.github.com/users/friso/orgs","repos_url":"https://api.github.com/users/friso/repos","events_url":"https://api.github.com/users/friso/events{/privacy}","received_events_url":"https://api.github.com/users/friso/received_events","type":"User","site_admin":false},"created_at":"2018-04-20T13:51:28Z","updated_at":"2018-04-20T13:51:28Z","author_association":"COLLABORATOR","body":"I'm looking into the details on this a bit.\r\n\r\nAddition of metrics to Divolte can be separated into two concerns: 1) instrumentation and 2) reporting.\r\n\r\nFor the sake of supporting different deployment architectures and monitoring solutions, Divolte needs to support multiple reporting backends. This should ideally be based on a single instrumentation code path.\r\n\r\nTo cover a broad range of contemporary deployment scenarios, we should consider the following backends:\r\n- JMX\r\n- statsd (this enables support for some popular hosted metrics solutions, such as [Datadog](https://www.datadoghq.com/))\r\n- Prometheus (natively, without push gateway)\r\n- logging / console / local csv writing (for local debugging and testing purposes these make sense)\r\n\r\nSeveral (collections of) libraries for both instrumentation and reporting exist, with overlapping functionalities. The two most serious contenders appear to be:\r\n- [Prometheus' official instrumentation library for the JVM](https://github.com/prometheus/client_java#included-collectors)\r\n- [Metrics](http://metrics.dropwizard.io/4.0.0/index.html)\r\n\r\nInstrumentation code with these libraries differs minimally. The notable challenging differences are:\r\n- The Prometheus library doesn't do JMX as backend\r\n- The Metrics library doesn't do Prometheus as backed (further complicated by the fact that Prometheus is pull based, whereas all of the supported backends are push based)\r\n- Prometheus' metric types overlap with those available in Metrics, but are not the same. Some mapping is required to go from one to the other.\r\n\r\nFor the sake of supporting JMX as first class, one implementation direction would be to instrument the Divolte code using Metrics and create a Prometheus compatible HTTP endpoint based on the collected metrics in Divolte. This is currently my preferred path. Doing the reverse of adding JMX reporting support to Prometheus' instrumentation library would probably require low level interactions with the MBean API, which I fear is time consuming.\r\n\r\nFinally, I want to develop at least a intuition and preferably some insights for the overhead of instrumentation both under normal load and under contention.\r\n\r\nI'll likely work on some of this first."}}
{"actor":{"display_login":"friso"}}