{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/kibana/issues/20393","repository_url":"https://api.github.com/repos/elastic/kibana","labels_url":"https://api.github.com/repos/elastic/kibana/issues/20393/labels{/name}","comments_url":"https://api.github.com/repos/elastic/kibana/issues/20393/comments","events_url":"https://api.github.com/repos/elastic/kibana/issues/20393/events","html_url":"https://github.com/elastic/kibana/issues/20393","id":337692107,"node_id":"MDU6SXNzdWUzMzc2OTIxMDc=","number":20393,"title":"Provide a Kibana Stats API that can be consumed by a Kibana Metricbeat metricset giving 1:1 parity with current Kibana Monitoring data","user":{"login":"tsullivan","id":908371,"node_id":"MDQ6VXNlcjkwODM3MQ==","avatar_url":"https://avatars1.githubusercontent.com/u/908371?v=4","gravatar_id":"","url":"https://api.github.com/users/tsullivan","html_url":"https://github.com/tsullivan","followers_url":"https://api.github.com/users/tsullivan/followers","following_url":"https://api.github.com/users/tsullivan/following{/other_user}","gists_url":"https://api.github.com/users/tsullivan/gists{/gist_id}","starred_url":"https://api.github.com/users/tsullivan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsullivan/subscriptions","organizations_url":"https://api.github.com/users/tsullivan/orgs","repos_url":"https://api.github.com/users/tsullivan/repos","events_url":"https://api.github.com/users/tsullivan/events{/privacy}","received_events_url":"https://api.github.com/users/tsullivan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-02T23:06:29Z","updated_at":"2018-07-02T23:16:53Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Ingest Team wants to be able to build a Kibana Metricbeat metricset that can call a Kibana HTTP API and receive the same data that gets stored in the `.monitoring-kibana-*` indices as it exists today.\r\n\r\nThe data stored in `.monitoring-kibana-*` comes from multiple sources, and gets compiled into 2 types:\r\n- `kibana_stats`:\r\n  - **Kibana ops stats** server uptime, system load, memory, etc *1\r\n  - **Kibana usage stats**: stats from saved objects: number of Kibana dashboards, saved searches, visualizations, index patterns, Timelion worksheets, Graph Worksheets *2\r\n  - **Reporting usage stats**: number of generated reports in the `.reporting-*` history *3\r\n- `kibana_settings`:\r\n  - **Advanced Settings**: admin email address, which is needed for Monitoring cluster alerts. This particular setting might not be needed for long, but other settings might come in later *4\r\n\r\nThe first milestone for the Metricbeat metricset will be to index Kibana-related data in Elasticsearch in the same format as it has been collected internally by Kibana. This will ensure that the current Monitoring UI and Telemetry functionality continue to work as-is without requiring any code changes.\r\n\r\nThe data includes information about X-Pack plugins (Graph and Reporting) and therefore the API should be implemented in X-Pack Main rather than OSS (core Kibana). The module will gather the same data in the same format that `BulkUploader` currently produces in the `kibana_monitoring` module of the Monitoring x-pack plugin.\r\n\r\nThere will not be a `usage` or `settings` API in core Kibana. There will continue to be a `stats` API in core Kibana featuring the \"Kibana ops stats\" provided for health checking the Kibana server.\r\n\r\n*1 `x-pack/plugins/monitoring/server/kibana_monitoring/collectors/get_ops_stats_collector.js`\r\n*2 `x-pack/plugins/monitoring/server/kibana_monitoring/collectors/get_kibana_usage_collector.js` \r\n*3 `x-pack/plugins/reporting/server/usage/get_reporting_usage_collector.js`\r\n*4 `x-pack/plugins/monitoring/server/kibana_monitoring/collectors/get_settings_collector.js`"},"comment":{"url":"https://api.github.com/repos/elastic/kibana/issues/comments/401965503","html_url":"https://github.com/elastic/kibana/issues/20393#issuecomment-401965503","issue_url":"https://api.github.com/repos/elastic/kibana/issues/20393","id":401965503,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTk2NTUwMw==","user":{"login":"ycombinator","id":51061,"node_id":"MDQ6VXNlcjUxMDYx","avatar_url":"https://avatars2.githubusercontent.com/u/51061?v=4","gravatar_id":"","url":"https://api.github.com/users/ycombinator","html_url":"https://github.com/ycombinator","followers_url":"https://api.github.com/users/ycombinator/followers","following_url":"https://api.github.com/users/ycombinator/following{/other_user}","gists_url":"https://api.github.com/users/ycombinator/gists{/gist_id}","starred_url":"https://api.github.com/users/ycombinator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ycombinator/subscriptions","organizations_url":"https://api.github.com/users/ycombinator/orgs","repos_url":"https://api.github.com/users/ycombinator/repos","events_url":"https://api.github.com/users/ycombinator/events{/privacy}","received_events_url":"https://api.github.com/users/ycombinator/received_events","type":"User","site_admin":false},"created_at":"2018-07-02T23:16:53Z","updated_at":"2018-07-02T23:16:53Z","author_association":"CONTRIBUTOR","body":"FYI: @ruflin @chrisronline @pickypg This issue came about after @tsullivan and I had a chat over Zoom about the best path forward to take the first baby step towards Metricbeat monitoring Kibana.\r\n\r\nOnce the HTTP API described by this issue is in place, I was thinking of creating an `xpack_stats` metricset in the Kibana module in Metricbeat that will call this API and ship the resulting data (after some parsing/normalization if necessary) directly into `.monitoring-kibana-mb-%{YYYY-MM-DD}` indices using the regular bulk API in ES.\r\n\r\nIn later stages, we can talk about and implement things like: no longer shipping settings (or at least ones requires by Cluster Alerts today, once that functionality moves completely into Kibana), changing the structure of documents to match ECS, changing index names, etc. For now the goal is simply to change the shipping path from custom Kibana code + custom Monitoring bulk endpoint in ES to Metricbeat + regular bulk endpoint in ES."}}
{"actor":{"display_login":"ycombinator"}}