{"action":"created","issue":{"url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/5925","repository_url":"https://api.github.com/repos/Graylog2/graylog2-server","labels_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/5925/labels{/name}","comments_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/5925/comments","events_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/5925/events","html_url":"https://github.com/Graylog2/graylog2-server/issues/5925","id":439670444,"node_id":"MDU6SXNzdWU0Mzk2NzA0NDQ=","number":5925,"title":"Improvement: Ability to send system logs into ES index","user":{"login":"gimmic","id":568957,"node_id":"MDQ6VXNlcjU2ODk1Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/568957?v=4","gravatar_id":"","url":"https://api.github.com/users/gimmic","html_url":"https://github.com/gimmic","followers_url":"https://api.github.com/users/gimmic/followers","following_url":"https://api.github.com/users/gimmic/following{/other_user}","gists_url":"https://api.github.com/users/gimmic/gists{/gist_id}","starred_url":"https://api.github.com/users/gimmic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gimmic/subscriptions","organizations_url":"https://api.github.com/users/gimmic/orgs","repos_url":"https://api.github.com/users/gimmic/repos","events_url":"https://api.github.com/users/gimmic/events{/privacy}","received_events_url":"https://api.github.com/users/gimmic/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2019-05-02T16:07:09Z","updated_at":"2019-05-03T08:49:05Z","closed_at":null,"author_association":"NONE","body":"I've started to notice signal-to-noise issues with the system overview page's centralized node logs. Every input change or general graylog cluster change results in at least 6 pages of repeating logs as each node does the same action. This makes paging through looking for actual information irksome. \r\n\r\n## Expected Behavior\r\nThere should be an option to tee off the graylog system logs to the backend ES in a dedicated index set. \r\n\r\n## Current Behavior\r\nGraylog system logs are stored in mongodb.\r\n35 graylog nodes * 3 messages per input change, deleting for example: STOPPING / STOPPED / TERMINATED = 105 log events when I delete an input, resulting in pages and pages of that event repeating.\r\n![image](https://user-images.githubusercontent.com/568957/57089285-be649580-6cc9-11e9-885a-8d87bb9094cd.png)\r\n\r\n\r\n## Possible Solution\r\nI understand it could be problematic to ONLY store system logs into ES in the event the backend ES cluster is having problems. Potentially use mongodb as primary and also index into a dedicated index for graylog_system, allowing you to use the graylog ecosystem to explore its own logs. \r\n\r\nAnother option would be to summarize identical events across the cluster as something like: \r\n> 2019-05-02T10:31:46-05:00 [35 nodes] Input [Raw/Plaintext TCP/5ccb0d543b3e1e7e1bbeb274] is now STARTING\r\n\r\n\r\n\r\n\r\n## Steps to Reproduce (for bugs)\r\nMake a lot of graylog nodes. Do stuff. Look at the system overview page. \r\n\r\n## Context\r\nI routinely check for things like indexes rollover rate and the easiest location to eyeball that is in the system overview page. It might be nice to also be able to use all the graylog feature set to review its own logs. Long term storage of graylog status logs also might not be best done in the configuration mongodb.\r\n\r\n* Graylog Version: 3.0.1"},"comment":{"url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/comments/489014689","html_url":"https://github.com/Graylog2/graylog2-server/issues/5925#issuecomment-489014689","issue_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/5925","id":489014689,"node_id":"MDEyOklzc3VlQ29tbWVudDQ4OTAxNDY4OQ==","user":{"login":"dennisoelkers","id":41929,"node_id":"MDQ6VXNlcjQxOTI5","avatar_url":"https://avatars1.githubusercontent.com/u/41929?v=4","gravatar_id":"","url":"https://api.github.com/users/dennisoelkers","html_url":"https://github.com/dennisoelkers","followers_url":"https://api.github.com/users/dennisoelkers/followers","following_url":"https://api.github.com/users/dennisoelkers/following{/other_user}","gists_url":"https://api.github.com/users/dennisoelkers/gists{/gist_id}","starred_url":"https://api.github.com/users/dennisoelkers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dennisoelkers/subscriptions","organizations_url":"https://api.github.com/users/dennisoelkers/orgs","repos_url":"https://api.github.com/users/dennisoelkers/repos","events_url":"https://api.github.com/users/dennisoelkers/events{/privacy}","received_events_url":"https://api.github.com/users/dennisoelkers/received_events","type":"User","site_admin":false},"created_at":"2019-05-03T08:49:05Z","updated_at":"2019-05-03T08:49:05Z","author_association":"MEMBER","body":"We were hesitating to store system/Graylog logs in ES to prevent feedback loops. We are thinking about making the system logs in mongodb available for analytics in the same was as we do with the ES indices."}}
{"actor":{"display_login":"dennisoelkers"}}