{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/beats/issues/1600","repository_url":"https://api.github.com/repos/elastic/beats","labels_url":"https://api.github.com/repos/elastic/beats/issues/1600/labels{/name}","comments_url":"https://api.github.com/repos/elastic/beats/issues/1600/comments","events_url":"https://api.github.com/repos/elastic/beats/issues/1600/events","html_url":"https://github.com/elastic/beats/issues/1600","id":153933570,"number":1600,"title":"Registry state persistence and cleanup","user":{"login":"ruflin","id":244900,"avatar_url":"https://avatars.githubusercontent.com/u/244900?v=3","gravatar_id":"","url":"https://api.github.com/users/ruflin","html_url":"https://github.com/ruflin","followers_url":"https://api.github.com/users/ruflin/followers","following_url":"https://api.github.com/users/ruflin/following{/other_user}","gists_url":"https://api.github.com/users/ruflin/gists{/gist_id}","starred_url":"https://api.github.com/users/ruflin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ruflin/subscriptions","organizations_url":"https://api.github.com/users/ruflin/orgs","repos_url":"https://api.github.com/users/ruflin/repos","events_url":"https://api.github.com/users/ruflin/events{/privacy}","received_events_url":"https://api.github.com/users/ruflin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/beats/labels/discuss","name":"discuss","color":"ffa500"},{"url":"https://api.github.com/repos/elastic/beats/labels/Filebeat","name":"Filebeat","color":"008000"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":19,"created_at":"2016-05-10T06:36:18Z","updated_at":"2016-07-11T13:11:19Z","closed_at":null,"body":"Filebeat currently lacks an option to cleanup the registry file. The goal of this issue is two answer three questions:\r\n\r\n* Which files are ignore for crawling?\r\n* Which states are removed from the registry?\r\n* How long is the state kept for files that cannot be found anymore?\r\n\r\nI suggest to introduce two new config options `clean_older` and `clean_removed` with the following behaviour and defaults. This is option 4 below.\r\n\r\n\r\n| Variable      | Description                                                                                                                          | Default  |\r\n|---------------|--------------------------------------------------------------------------------------------------------------------------------------|----------|\r\n| close_older   | Closes file handler for files which are older then x                                                                                 | 1h       |\r\n| clean_removed | Cleans the registry file from files which are not found anymore after x                                                              | 24h      |\r\n| ignore_older  | Does not crawl files which are older then ignore_older. Still keeps the file state and moves the file offset to the end of the file. | infinity |\r\n| clean_older   | Removes file from the registry which are older then x                                                                                | infinity |\r\n\r\nBetter name suggestions for the new variables are welcome.\r\n\r\n# Option Details\r\n\r\nBelow is the current behaviour and the different options described in more detail.\r\n\r\n## Current Behaviour\r\n\r\nThe current behaviour is as following:\r\n\r\n* File offset for files reaching `ignore_older` are set to the end of the file and persisted.\r\n\r\nThis brings the problem that the registry is never cleaned up.\r\n\r\n\r\n## Option 1 - ignore_older double meaning\r\n\r\nOne option is to use ignore_older to also cleanup the state. This would have the following consequences:\r\n\r\n* If a file reaches ignore_older, the state is removed from the registry\r\n* That means if a file that reached ignore_older is updated again, it will be read from the beginning as no state was stored\r\n\r\n\r\n\r\n## Option 2 - clean_older >= ignore_older\r\n\r\nThe introduction of a `clean_older` variable would allow to set a time when the registry should be cleaned up. In case ignore_older has the same behaviour as now, it requires that clean_older is >= ignore_older as otherwise the two would get into a race condition. As ignore_older is set by default to infinity, it would also mean that clean_older is disabled by default. If clean_older is enabled with a time, it would mean also ignore_older has to be set.\r\n\r\n\r\n## Option 3 - ignore_older, clean_removed\r\n\r\nIntroducing a variable `clean_removed` would allow to enable an option, that only files which disappear from disk are cleaned up. `clean_removed` could either be a bool and files that disappear would be directly removed or it could also be time a duration after whicht the files are removed. This means all files which still exist will be kept in the registry.\r\n\r\nIn case clean_removed is duration, it would require `clean_removed >= close_older` or `force_close_files` to be enable to make sure the files are closed.\r\n\r\n\r\n## Option 4 - clean_older >= ignore_older, clean_older >= clean_removed\r\n\r\nOption 4 would be combining all these options. This is my preferred option as it gives the full flexibility and keeps the behaviour of the existing variables."},"comment":{"url":"https://api.github.com/repos/elastic/beats/issues/comments/231729549","html_url":"https://github.com/elastic/beats/issues/1600#issuecomment-231729549","issue_url":"https://api.github.com/repos/elastic/beats/issues/1600","id":231729549,"user":{"login":"ruflin","id":244900,"avatar_url":"https://avatars.githubusercontent.com/u/244900?v=3","gravatar_id":"","url":"https://api.github.com/users/ruflin","html_url":"https://github.com/ruflin","followers_url":"https://api.github.com/users/ruflin/followers","following_url":"https://api.github.com/users/ruflin/following{/other_user}","gists_url":"https://api.github.com/users/ruflin/gists{/gist_id}","starred_url":"https://api.github.com/users/ruflin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ruflin/subscriptions","organizations_url":"https://api.github.com/users/ruflin/orgs","repos_url":"https://api.github.com/users/ruflin/repos","events_url":"https://api.github.com/users/ruflin/events{/privacy}","received_events_url":"https://api.github.com/users/ruflin/received_events","type":"User","site_admin":false},"created_at":"2016-07-11T13:11:19Z","updated_at":"2016-07-11T13:11:19Z","body":"As a note: The general discussion came up to change `prospectors` to `inputs` and `harvester` to `reader`."}}
{"actor":{"display_login":"ruflin"}}