{"action":"created","issue":{"url":"https://api.github.com/repos/mholt/caddy/issues/577","repository_url":"https://api.github.com/repos/mholt/caddy","labels_url":"https://api.github.com/repos/mholt/caddy/issues/577/labels{/name}","comments_url":"https://api.github.com/repos/mholt/caddy/issues/577/comments","events_url":"https://api.github.com/repos/mholt/caddy/issues/577/events","html_url":"https://github.com/mholt/caddy/issues/577","id":131501596,"number":577,"title":"Feedback/Discussion: Stats","user":{"login":"captncraig","id":192540,"avatar_url":"https://avatars.githubusercontent.com/u/192540?v=3","gravatar_id":"","url":"https://api.github.com/users/captncraig","html_url":"https://github.com/captncraig","followers_url":"https://api.github.com/users/captncraig/followers","following_url":"https://api.github.com/users/captncraig/following{/other_user}","gists_url":"https://api.github.com/users/captncraig/gists{/gist_id}","starred_url":"https://api.github.com/users/captncraig/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/captncraig/subscriptions","organizations_url":"https://api.github.com/users/captncraig/orgs","repos_url":"https://api.github.com/users/captncraig/repos","events_url":"https://api.github.com/users/captncraig/events{/privacy}","received_events_url":"https://api.github.com/users/captncraig/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mholt/caddy/labels/addon","name":"addon","color":"207de5"},{"url":"https://api.github.com/repos/mholt/caddy/labels/discussion","name":"discussion","color":"006b75"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-02-04T23:07:01Z","updated_at":"2016-02-08T21:05:53Z","closed_at":null,"body":"I've started an implementation of stats, but wanted to gather feedback before I get too much deeper.\r\n\r\n## why not [expvar](https://github.com/mholt/caddy/pull/285)?\r\n\r\nexpvar is nice for ad-hoc data, but I find it is not the best for structured metrics. If you want to export to a time series database you really need a format more tailored for the use case than arbitrary json.\r\n\r\n## go-metrics\r\n\r\n[go-metrics](https://github.com/rcrowley/go-metrics) is a widely used go package for tracking metrics. If we use this package to store metrics, any caaddy package (core or plugin) can add metrics to the default registry and they will be accessible for output.\r\nThe only package anybody needs to reference directly is go-metrics.\r\n\r\nIt can support major metric types we would need:\r\n\r\n- Counters for ever-increasing values (number of requests)\r\n- Samples for time data (request time)\r\n- Gauges for momentary data (memory / goroutines)\r\n\r\n## Push Or Pull?\r\n\r\n### both!\r\n\r\nOn a configurable timer, the stats module will snapshot all registered metrics into a data structure:\r\n\r\n```\r\ntype Measurement struct{\r\n   Metric string // metric name (caddy.request_count)\r\n   Tags map[string]string // host=myserver, server=foo.com, path=getUser\r\n   Values map[string] int64 // Can have multiple values per measurement (Sum,99th,95th,Count), or just a single \"Value\". Handled differently by each tsdb.\r\n   Timestamp time.Time    \r\n}\r\n```\r\n\r\na web endpoint can be set up to display json of the latest snapshot, or we can send it to time series databases by implementing pushers for influx, graphite, opentsdb, prometheus, ...\r\n\r\n`type Pusher interface{ Send(batch []Measurement) error }`\r\n\r\n## Tags\r\n\r\nI believe in multi-dimensional metrics. Almost every tsdb supports tagged metrics, or can be adapted from the tag model. go-metrics doesn't directly support tags in the registry, but since it is just keyed on\r\na name we can encode tags in the metric name: `caddy.request_count{server=foo.com,path=getUser}`, or `caddy.status_codes{server=foo.com,code=200}`. \r\n\r\n## Stat Module\r\n\r\nThe stat module will have two main purposes:\r\n\r\n1. Sit toward the front of the middleware chain and collect request counts, timings, statuses, error counts, etc.\r\n2. Gather metrics and send to tsdbs, as well as host raw data for pull consumers.\r\n\r\nJust adding `stats` as a directive should be enough to get a server collecting stats. One or more server blocks can add additional fields:\r\n\r\n```\r\nstats /stats { #can pull data from /stats\r\n    interval 15s #global, should only be defined once\r\n    send influx https://myInfluxServer:8086  \r\n}    \r\n```\r\n\r\n## Metrics to collect\r\n\r\nInitially, the stat module will collect:\r\n\r\n- `caddy.requests{server=foo.com,path=/}` - counter of total requests by server and path.\r\n- `caddy.status_codes{code=200,server=foo.com,path=/}` - conter of total requests by response code, server and path\r\n- `caddy.errors{server=foo.com,path=/}` - total errors by server and path\r\n- `caddy.response_time{server=foo.com,path=/}` - timings per server/path. Will include count,avg,99th,95th,stdDev,etc.\r\n- server uptime, basic mem and gc stats, goroutine count, etc.\r\n\r\nOther possibilities include:\r\n\r\n- proxy stats and timings per upstream\r\n- byte throughputs from server core\r\n- https certificate lifetime information\r\n- ip block counts\r\n\r\n### A note on paths:\r\n\r\nIn the above examples I grouped by \"path\". What people consider a group of paths they want to measyure could vary wildly. Default is always `/` if e can't match anything.\r\nOther rules may be given to match a common path or method or maybe more.\r\n\r\n`path /foo foo #alias /foo* as foo`\r\n\r\n`path /user GET getUser`\r\n`path /user PUT,POST,PATCH saveUser`\r\n\r\nCan we reuse any match logic from other directives?\r\n\r\nIf other directives below me on the stack want to add metrics a way to pass context up and down the stack might be really useful so I can communicate the path to them.\r\n\r\n## Visualization\r\n\r\nMy initial intent is to rely on external facilities for graphing and such. I can share some common grafana dashboards for useful things from various data sources.\r\n\r\nDown the road we may want a built-in data page with graphs and such, either from the external db or from some kind of embedded cache of the data. Not sure what that would be.\r\n\r\n### Feedback?\r\n\r\nI feel like this is a good direction to get what I want out of caddy, but I would love feedback for how others would want to use it. Am I missing something big? Is there a different direction this could/should go?"},"comment":{"url":"https://api.github.com/repos/mholt/caddy/issues/comments/181568073","html_url":"https://github.com/mholt/caddy/issues/577#issuecomment-181568073","issue_url":"https://api.github.com/repos/mholt/caddy/issues/577","id":181568073,"user":{"login":"captncraig","id":192540,"avatar_url":"https://avatars.githubusercontent.com/u/192540?v=3","gravatar_id":"","url":"https://api.github.com/users/captncraig","html_url":"https://github.com/captncraig","followers_url":"https://api.github.com/users/captncraig/followers","following_url":"https://api.github.com/users/captncraig/following{/other_user}","gists_url":"https://api.github.com/users/captncraig/gists{/gist_id}","starred_url":"https://api.github.com/users/captncraig/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/captncraig/subscriptions","organizations_url":"https://api.github.com/users/captncraig/orgs","repos_url":"https://api.github.com/users/captncraig/repos","events_url":"https://api.github.com/users/captncraig/events{/privacy}","received_events_url":"https://api.github.com/users/captncraig/received_events","type":"User","site_admin":false},"created_at":"2016-02-08T21:05:53Z","updated_at":"2016-02-08T21:05:53Z","body":"Yeah, part of the design is that proxy can add stats per backend and stuff and it will just show up; and won't cause problems if  stats is no loaded."}}