{"action":"created","issue":{"url":"https://api.github.com/repos/theforeman/foreman/issues/5096","repository_url":"https://api.github.com/repos/theforeman/foreman","labels_url":"https://api.github.com/repos/theforeman/foreman/issues/5096/labels{/name}","comments_url":"https://api.github.com/repos/theforeman/foreman/issues/5096/comments","events_url":"https://api.github.com/repos/theforeman/foreman/issues/5096/events","html_url":"https://github.com/theforeman/foreman/pull/5096","id":282877523,"number":5096,"title":"Fixes #18675 - telemetry foreman API","user":{"login":"lzap","id":49752,"avatar_url":"https://avatars0.githubusercontent.com/u/49752?v=4","gravatar_id":"","url":"https://api.github.com/users/lzap","html_url":"https://github.com/lzap","followers_url":"https://api.github.com/users/lzap/followers","following_url":"https://api.github.com/users/lzap/following{/other_user}","gists_url":"https://api.github.com/users/lzap/gists{/gist_id}","starred_url":"https://api.github.com/users/lzap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lzap/subscriptions","organizations_url":"https://api.github.com/users/lzap/orgs","repos_url":"https://api.github.com/users/lzap/repos","events_url":"https://api.github.com/users/lzap/events{/privacy}","received_events_url":"https://api.github.com/users/lzap/received_events","type":"User","site_admin":false},"labels":[{"id":79040689,"url":"https://api.github.com/repos/theforeman/foreman/labels/Needs%20testing","name":"Needs testing","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2017-12-18T13:24:03Z","updated_at":"2018-02-14T10:59:13Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/theforeman/foreman/pulls/5096","html_url":"https://github.com/theforeman/foreman/pull/5096","diff_url":"https://github.com/theforeman/foreman/pull/5096.diff","patch_url":"https://github.com/theforeman/foreman/pull/5096.patch"},"body":"Foreman telemetry core API. This PR brings a common telemetry API which provides three generic metric types:\r\n\r\n* **counter** (monotony increasing counter - for example number of http requests)\r\n* **gauge** (arbitrary float value - for example number of jobs queued)\r\n* **histogram** (arbitrary amount of buckets with observations and float values - for example duration of a request)\r\n\r\nEach individual metric must be predefined in config/initializers/5_telemetry_metrics.rb file in this format:\r\n\r\n    telemetry.add_counter(:http_requests, 'A counter of HTTP requests made', [:controller, :action])\r\n\r\nOptionally buckets can be defined for histograms (a sane default is used if not defined):\r\n\r\n    telemetry.add_histogram(:http_request_total_duration, 'Total duration of controller action', [:controller, :action], [5, 10, 50, 200, 1000])\r\n\r\nFor more about why I picked these, watch my proposal video: https://www.youtube.com/watch?v=gCLSI9-4QpE\r\n\r\nThere are two implementations for the best possible integration options: Prometheus endpoint and Statsd packets. To configure the former, add this to your settings:\r\n\r\n```yaml\r\n:telemetry:\r\n  :type: 'prometheus'\r\n  :prefix: 'fm_rails'\r\n```\r\n\r\nThen visit `/metrics` to see aggregated data. Warning - due to limitation of Prometheus Ruby library, this will not work for multi-process web servers (our case) and this will provide you incorrect data. Therefore, until this is fixed upstream, a recommended way is to use Statsd:\r\n\r\n```yaml\r\n:telemetry:\r\n  :type: 'statsd'\r\n  :prefix: 'fm_rails'\r\n  :statsd:\r\n    :host: '127.0.0.1:8125'\r\n    :protocol: 'statsd'\r\n```\r\n\r\nThen download and run any statsd aggregator on localhost port 8125. I tested with statsite and statsd_exporter for Prometheus, which is super easy to use:\r\n\r\n```\r\n./statsd_exporter -statsd.listen-udp :8125\r\n```\r\n\r\nThis PR provides a rake task that will create useful mapping for statsd_exporter which makes life much more easier and provides label mapping:\r\n\r\n\r\n```\r\nbe rake telemetry:prometheus_statsd | tee /tmp/mapping.yaml\r\n./statsd_exporter -statsd.listen-udp :8125 -statsd.mapping-config /tmp/mapping.yaml\r\n```\r\n\r\nThe last step is to grab the data into some monitoring framework, Prometheus is very easy to use as it's a single binary and configuration is easy:\r\n\r\n```\r\nscrape_configs:\r\n  - job_name: 'foreman'\r\n    static_configs:\r\n      - targets: ['localhost:9102']\r\n\r\n./prometheus\r\n```\r\n\r\nThis PR adds bunch of basic telemetry data, we can add more later on via initializer, this could be DSL instead of direct Ruby code (config/initializers/5_telemetry_metrics.rb). The PR currently adds:\r\n\r\n| Metric name | Labels | Type | Description |\r\n| ----------- | ------ | ---- | ----------- |\r\n| fm_rails_activerecord_instances | class | counter | Number of instances of ActiveRecord models |\r\n| fm_rails_bruteforce_locked_ui_logins |  | counter | Number of blocked logins via bruteforce protection |\r\n| fm_rails_failed_ui_logins |  | counter | Number of failed logins in total |\r\n| fm_rails_http_request_db_duration | controller,action | histogram | Time spent in database for a request |\r\n| fm_rails_http_request_total_duration | controller,action | histogram | Total duration of controller action |\r\n| fm_rails_http_request_view_duration | controller,action | histogram | Time spent in view for a request |\r\n| fm_rails_http_requests | controller,action | counter | A counter of HTTP requests made |\r\n| fm_rails_proxy_api_duration | method | histogram | Time spent waiting for Proxy (ms) |\r\n| fm_rails_proxy_api_response_code | code | counter | Number of Proxy API responses per HTTP code |\r\n| fm_rails_ruby_gc_allocated_objects | controller,action | counter | Ruby GC statistics per request (total_allocated_objects) |\r\n| fm_rails_ruby_gc_count | controller,action | counter | Ruby GC statistics per request (count) |\r\n| fm_rails_ruby_gc_freed_objects | controller,action | counter | Ruby GC statistics per request (total_freed_objects) |\r\n| fm_rails_ruby_gc_major_count | controller,action | counter | Ruby GC statistics per request (major_gc_count) |\r\n| fm_rails_ruby_gc_minor_count | controller,action | counter | Ruby GC statistics per request (minor_gc_count) |\r\n| fm_rails_successful_ui_logins |  | counter | Number of successful logins in total |\r\n\r\nA short demo how to set thigs up:\r\n\r\nhttps://youtu.be/i5iCOLEByZk\r\n\r\nTODO:\r\n\r\n* [ ] Installer\r\n* [ ] Documentation\r\n* [ ] Packages for new dependencies\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/theforeman/foreman/issues/comments/365568509","html_url":"https://github.com/theforeman/foreman/pull/5096#issuecomment-365568509","issue_url":"https://api.github.com/repos/theforeman/foreman/issues/5096","id":365568509,"user":{"login":"dLobatog","id":598891,"avatar_url":"https://avatars2.githubusercontent.com/u/598891?v=4","gravatar_id":"","url":"https://api.github.com/users/dLobatog","html_url":"https://github.com/dLobatog","followers_url":"https://api.github.com/users/dLobatog/followers","following_url":"https://api.github.com/users/dLobatog/following{/other_user}","gists_url":"https://api.github.com/users/dLobatog/gists{/gist_id}","starred_url":"https://api.github.com/users/dLobatog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dLobatog/subscriptions","organizations_url":"https://api.github.com/users/dLobatog/orgs","repos_url":"https://api.github.com/users/dLobatog/repos","events_url":"https://api.github.com/users/dLobatog/events{/privacy}","received_events_url":"https://api.github.com/users/dLobatog/received_events","type":"User","site_admin":false},"created_at":"2018-02-14T10:59:13Z","updated_at":"2018-02-14T10:59:13Z","author_association":"MEMBER","body":"Due to the recent obal changes in packaging, I'll hold off merging until that's resolved - shouldn't take a long time"}}
{"actor":{"display_login":"dLobatog"}}