{"action":"created","issue":{"url":"https://api.github.com/repos/joyent/containerpilot/issues/27","repository_url":"https://api.github.com/repos/joyent/containerpilot","labels_url":"https://api.github.com/repos/joyent/containerpilot/issues/27/labels{/name}","comments_url":"https://api.github.com/repos/joyent/containerpilot/issues/27/comments","events_url":"https://api.github.com/repos/joyent/containerpilot/issues/27/events","html_url":"https://github.com/joyent/containerpilot/issues/27","id":120300011,"number":27,"title":"Proposal: add user defined application telemetry","user":{"login":"misterbisson","id":1709697,"avatar_url":"https://avatars.githubusercontent.com/u/1709697?v=3","gravatar_id":"","url":"https://api.github.com/users/misterbisson","html_url":"https://github.com/misterbisson","followers_url":"https://api.github.com/users/misterbisson/followers","following_url":"https://api.github.com/users/misterbisson/following{/other_user}","gists_url":"https://api.github.com/users/misterbisson/gists{/gist_id}","starred_url":"https://api.github.com/users/misterbisson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/misterbisson/subscriptions","organizations_url":"https://api.github.com/users/misterbisson/orgs","repos_url":"https://api.github.com/users/misterbisson/repos","events_url":"https://api.github.com/users/misterbisson/events{/privacy}","received_events_url":"https://api.github.com/users/misterbisson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/joyent/containerpilot/labels/enhancement","name":"enhancement","color":"71b8c7"},{"url":"https://api.github.com/repos/joyent/containerpilot/labels/released","name":"released","color":"bfe5bf"}],"state":"closed","locked":false,"assignee":{"login":"tgross","id":1409219,"avatar_url":"https://avatars.githubusercontent.com/u/1409219?v=3","gravatar_id":"","url":"https://api.github.com/users/tgross","html_url":"https://github.com/tgross","followers_url":"https://api.github.com/users/tgross/followers","following_url":"https://api.github.com/users/tgross/following{/other_user}","gists_url":"https://api.github.com/users/tgross/gists{/gist_id}","starred_url":"https://api.github.com/users/tgross/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgross/subscriptions","organizations_url":"https://api.github.com/users/tgross/orgs","repos_url":"https://api.github.com/users/tgross/repos","events_url":"https://api.github.com/users/tgross/events{/privacy}","received_events_url":"https://api.github.com/users/tgross/received_events","type":"User","site_admin":false},"milestone":null,"comments":33,"created_at":"2015-12-03T23:55:24Z","updated_at":"2016-04-12T15:43:44Z","closed_at":"2016-04-12T15:43:43Z","body":"This is poorly thought through so far, but I wanted to open up discussion for it.\r\n\r\nHealth checking gives us a binary way to determine if the app is working or not. If the app is not healthy, we should not send any requests to it and we most likely need to spawn a replacement. But scaling depends on more than binary app health. Every app has one or more performance indicators that can be used to determine if the app is nearing overload and should be scaled up or is too lightly loaded and should be scaled down. In the spirit of what Containerbuddy does to make applications container-native, I think it might also make sense to add an awareness of those performance indicators.\r\n\r\nSome performance indicators can be read from the system. The [five minute load average](https://en.wikipedia.org/wiki/Load_%28computing%29) reported by the kernel may work well for many apps, but many of the most interesting indicators come from the app itself.\r\n\r\nIn MySQL, the number of `Query` entries from [`SHOW PROCESSLIST`](http://dev.mysql.com/doc/refman/5.7/en/show-processlist.html) that are in any [`Waiting`](http://dev.mysql.com/doc/refman/5.7/en/general-thread-states.html) state can be a very significant performance indicator, and it's one that is best retrieved from within the container.\r\n\r\nIn Nginx, the average [request processing time](http://nginx.org/en/docs/http/ngx_http_core_module.html#var_request_time) is useful info, but that's only output in the logs, which are hopefully not inside the container (if they were, [ngxtop](https://github.com/lebinh/ngxtop) would be a nice tool to help understand them). But we do expect [`http_stub_status_module`](http://nginx.org/en/docs/http/ngx_http_stub_status_module.html) in our [triton-nginx image](https://github.com/tgross/triton-nginx), so we can look at that instead. `Active connections` vs. the [`worker_connections`](http://nginx.org/en/docs/ngx_core_module.html#worker_connections) limit is a hugely important number there. Any delta between `accepts` and `handled` is a huge red flag. Or, perhaps, the `Waiting` number is an inverse indicator (high numbers indicate low activity, low numbers are high activity, zero could be critical). (See https://github.com/tgross/triton-nginx/issues/3 for further musing on this.)\r\n\r\nHere's what I'm imagining it would look like in the config (though it would be obviously unlikely to actually mix Nginx and MySQL in a single image):\r\n\r\n```json\r\n  \"kpis\": [\r\n    {\r\n      \"name\": \"system-loadavg\",\r\n      \"poll\": 103,\r\n      \"kpi\": \"/opt/containerbuddy/system-loadavg.sh\"\r\n    },\r\n    {\r\n      \"name\": \"mysql-queries-waiting\",\r\n      \"poll\": 31,\r\n      \"kpi\": \"/my/bin/mysql-queries-waiting.sh\"\r\n    },\r\n    {\r\n      \"name\": \"nginx-busy\",\r\n      \"poll\": 17,\r\n      \"kpi\": \"/my/bin/nginx-busy.sh\"\r\n    },\r\n    {\r\n      \"name\": \"nginx-missed-connections\",\r\n      \"poll\": 137,\r\n      \"kpi\": \"/my/bin/nginx-missed-connections.sh\"\r\n    }\r\n  ]\r\n```\r\n\r\nIn order, the above four KPI entries would return:\r\n\r\n1. The 5 minute load average (the trailing average, returned every 103 seconds). This is decimal value. It should generally not be higher than the count of VCPUs the instance has access to, though that can be hard to figure out in a Docker environment.\r\n1. The count of MySQL `Query` entries from [`SHOW PROCESSLIST`](http://dev.mysql.com/doc/refman/5.7/en/show-processlist.html) that are in any [`Waiting`](http://dev.mysql.com/doc/refman/5.7/en/general-thread-states.html) state. `0` is great. `1` or above can be trouble. `10` or more is probably critical.\r\n1.  The result of Nginx's `(${Active_connections} - ${Waiting}) / ${worker_connections}`, a decimal value. `1` is maxed out, `0.5` is 50% busy.\r\n1. The result of Nginx's `accepts - handled` vars, an integer value. Anything other than `0` here is probably critical.\r\n\r\nThe executables that get these KPIs are to be provided by the app developer/packager, though it might make sense to have a common way to get thee system load average as part of Containerbuddy.\r\n\r\nAs with the health checks, the executables would be run periodically at the time specified. The health checks could then be returned in `stdout` as newline delimited JSON as they're executed, for interpretation and use by whatever tools are reading the logs.\r\n\r\nI wanted to offer those specific use cases as as an exercise to figure out what data would be handled here. In short, I'm thinking each KPI is a numeric (decimal allowed) time series value. The executable will simply return that value and Containerbuddy can pass it on with the name given in the config."},"comment":{"url":"https://api.github.com/repos/joyent/containerpilot/issues/comments/208969378","html_url":"https://github.com/joyent/containerpilot/issues/27#issuecomment-208969378","issue_url":"https://api.github.com/repos/joyent/containerpilot/issues/27","id":208969378,"user":{"login":"tgross","id":1409219,"avatar_url":"https://avatars.githubusercontent.com/u/1409219?v=3","gravatar_id":"","url":"https://api.github.com/users/tgross","html_url":"https://github.com/tgross","followers_url":"https://api.github.com/users/tgross/followers","following_url":"https://api.github.com/users/tgross/following{/other_user}","gists_url":"https://api.github.com/users/tgross/gists{/gist_id}","starred_url":"https://api.github.com/users/tgross/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgross/subscriptions","organizations_url":"https://api.github.com/users/tgross/orgs","repos_url":"https://api.github.com/users/tgross/repos","events_url":"https://api.github.com/users/tgross/events{/privacy}","received_events_url":"https://api.github.com/users/tgross/received_events","type":"User","site_admin":false},"created_at":"2016-04-12T15:43:43Z","updated_at":"2016-04-12T15:43:43Z","body":"Done and included in the 2.0 release: https://github.com/joyent/containerpilot/releases/tag/2.0.0"}}