{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/beats/issues/7935","repository_url":"https://api.github.com/repos/elastic/beats","labels_url":"https://api.github.com/repos/elastic/beats/issues/7935/labels{/name}","comments_url":"https://api.github.com/repos/elastic/beats/issues/7935/comments","events_url":"https://api.github.com/repos/elastic/beats/issues/7935/events","html_url":"https://github.com/elastic/beats/issues/7935","id":349244928,"node_id":"MDU6SXNzdWUzNDkyNDQ5Mjg=","number":7935,"title":"Support for ILM in Beats","user":{"login":"ruflin","id":244900,"node_id":"MDQ6VXNlcjI0NDkwMA==","avatar_url":"https://avatars3.githubusercontent.com/u/244900?v=4","gravatar_id":"","url":"https://api.github.com/users/ruflin","html_url":"https://github.com/ruflin","followers_url":"https://api.github.com/users/ruflin/followers","following_url":"https://api.github.com/users/ruflin/following{/other_user}","gists_url":"https://api.github.com/users/ruflin/gists{/gist_id}","starred_url":"https://api.github.com/users/ruflin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ruflin/subscriptions","organizations_url":"https://api.github.com/users/ruflin/orgs","repos_url":"https://api.github.com/users/ruflin/repos","events_url":"https://api.github.com/users/ruflin/events{/privacy}","received_events_url":"https://api.github.com/users/ruflin/received_events","type":"User","site_admin":false},"labels":[{"id":577798610,"node_id":"MDU6TGFiZWw1Nzc3OTg2MTA=","url":"https://api.github.com/repos/elastic/beats/labels/:Outputs","name":":Outputs","color":"ffffff","default":false},{"id":294278853,"node_id":"MDU6TGFiZWwyOTQyNzg4NTM=","url":"https://api.github.com/repos/elastic/beats/labels/libbeat","name":"libbeat","color":"008000","default":false},{"id":296601926,"node_id":"MDU6TGFiZWwyOTY2MDE5MjY=","url":"https://api.github.com/repos/elastic/beats/labels/meta","name":"meta","color":"ff8c00","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2018-08-09T18:43:51Z","updated_at":"2018-12-12T19:01:06Z","closed_at":null,"author_association":"COLLABORATOR","body":"With https://github.com/elastic/elasticsearch/issues/29823#issuecomment-384130831 index lifecycle management is added to Elasticsearch. ILM is especially useful in the case of Beats to automate rollover. This will ensure all indices have a similar size instead of having daily indices where size can vary based on how many Beats and the number of events sent.\r\n\r\nThis is a description of the possible implementation in Beats. It still might change during implementation.\r\n\r\n## Technical Implementation\r\n\r\nThe basic implementation if ILM is enabled in Beats will look as following:\r\n\r\nThe Beat template will contain the ILM policy which should be used:\r\n\r\n```\r\nPUT _template/filebeat-6.5.0\r\n{\r\n  \"index_patterns\": [\"filebeat-6.5.0-*\"],\r\n  \"settings\": {\r\n    \"index.lifecycle.name\": \"filebeat-policy\",\r\n    \"index.lifecycle.rollover_alias\": \"filebeat-6.5.0\"\r\n  },\r\n  \"mappings\": {\r\n    \"_doc\": {\r\n      ...\r\n    }\r\n  }\r\n}\r\n````\r\n\r\nThis can already configured today through the following settings in Beats:\r\n\r\n```\r\nsetup.template.settings.index.lifecycle.name: \"filebeat-policy\"\r\nsetup.template.settings.index.lifecycle.rollover_alias: \"filebeat-6.5.0\"\r\n```\r\n\r\nWith ILM enabled these settings will be written automatically.\r\n\r\nAs soon as the template is loaded, the Beat will check for the existance of the write alias:\r\n\r\n```\r\nHEAD filebeat-6.5.0\r\n```\r\n\r\nIn case the write alias does not exist yet, it will be created:\r\n\r\n```\r\nPUT filebeat-6.5.0-000001\r\n{\r\n  \"aliases\": {\r\n      \"filebeat-6.5.0\":{\r\n            \"is_write_index\": true\r\n     }\r\n  }\r\n}\r\n```\r\n\r\nFrom here on all data is sent to the `filebeat-6.5.0` alias and things work like usual.\r\n\r\n## Configuration\r\n\r\nThe configuration of ILM belongs to the Elasticsearch output and could look as following.\r\n```\r\nelasticsearch.output.ilm:\r\n  enable: true\r\n  write_alias: filebeat\r\n  index: filebeat # Do we even want the version to be configurable? -> ES only and a lot of people break things with it\r\n  pattern: 000001\r\n  policy: filebeat-policy # What if the policy is also set in the settings?\r\n```\r\n\r\nThe special part in the above is that the Beat version was left out. A common issue in Beats is that the version number is sometimes remove from the index which can cause issues on migration. To prevent this issue for ILM the version is automatically added to the write alias, index names and the template. We could add an additional config like `automatic_version: false` to disable this feature if we want.\r\n\r\n\r\nQuestion:\r\n\r\n* Should ILM config be it's own top level entry instead of part of elasticsearch ouptut?\r\n\r\n## Example Policy\r\n\r\nAn example policy could look as following.\r\n\r\n```\r\nPUT _ilm/filebeat-policy\r\n{\r\n   \"policy\": {\r\n     \"type\": \"timeseries\",\r\n     \"phases\": {\r\n       \"hot\": {\r\n         \"after\": \"0s\",\r\n         \"actions\": {\r\n          \"rollover\": {\r\n            \"max_docs\": \"20\"\r\n          }\r\n         }\r\n       }\r\n     }\r\n   }\r\n}\r\n```\r\n\r\nThis will create a new alias every 20 documents. In a real world example larger numbers and other rollover criterias can be used.\r\n\r\nThe policy is expected to be loaded with `filebeat setup ilm-policy`. A policy can be loaded at any time and is not required on template generation or data ingestion.\r\n\r\n## Questions\r\n\r\n* Do we provide a default policy?\r\n* If yes, what is our default policy? Do we have different phases? Is it different per Beat?\r\n* What if Beats is started against an ES instance without ILM and ILM is configured? It should not start an error out.\r\n\r\n## Notes for testing\r\n\r\nILM policies are triggered every 10m by default. This can be changed as a cluster setting:\r\n\r\n```\r\nPUT /_cluster/settings\r\n{\r\n    \"persistent\" : {\r\n        \"indices.lifecycle.poll_interval\": \"5s\"\r\n    }\r\n}\r\n```\r\n\r\nThe above means policies are triggered every 5s."},"comment":{"url":"https://api.github.com/repos/elastic/beats/issues/comments/446703704","html_url":"https://github.com/elastic/beats/issues/7935#issuecomment-446703704","issue_url":"https://api.github.com/repos/elastic/beats/issues/7935","id":446703704,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjcwMzcwNA==","user":{"login":"clintongormley","id":56599,"node_id":"MDQ6VXNlcjU2NTk5","avatar_url":"https://avatars0.githubusercontent.com/u/56599?v=4","gravatar_id":"","url":"https://api.github.com/users/clintongormley","html_url":"https://github.com/clintongormley","followers_url":"https://api.github.com/users/clintongormley/followers","following_url":"https://api.github.com/users/clintongormley/following{/other_user}","gists_url":"https://api.github.com/users/clintongormley/gists{/gist_id}","starred_url":"https://api.github.com/users/clintongormley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clintongormley/subscriptions","organizations_url":"https://api.github.com/users/clintongormley/orgs","repos_url":"https://api.github.com/users/clintongormley/repos","events_url":"https://api.github.com/users/clintongormley/events{/privacy}","received_events_url":"https://api.github.com/users/clintongormley/received_events","type":"User","site_admin":false},"created_at":"2018-12-12T19:01:06Z","updated_at":"2018-12-12T19:01:06Z","author_association":"MEMBER","body":"How did we get to 25GB?  We've been recommending 50GB and benchmarking shows that that is still tiny.  Why are we recommending such tiny indices as a default?  I'd be leaning towards 100GB."}}
{"actor":{"display_login":"clintongormley"}}