{"action":"created","issue":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/192","repository_url":"https://api.github.com/repos/logstash-plugins/logstash-input-file","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/192/labels{/name}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/192/comments","events_url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/192/events","html_url":"https://github.com/logstash-plugins/logstash-input-file/pull/192","id":326125590,"number":192,"title":"Add better support for file rotation","user":{"login":"guyboertje","id":84912,"avatar_url":"https://avatars2.githubusercontent.com/u/84912?v=4","gravatar_id":"","url":"https://api.github.com/users/guyboertje","html_url":"https://github.com/guyboertje","followers_url":"https://api.github.com/users/guyboertje/followers","following_url":"https://api.github.com/users/guyboertje/following{/other_user}","gists_url":"https://api.github.com/users/guyboertje/gists{/gist_id}","starred_url":"https://api.github.com/users/guyboertje/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guyboertje/subscriptions","organizations_url":"https://api.github.com/users/guyboertje/orgs","repos_url":"https://api.github.com/users/guyboertje/repos","events_url":"https://api.github.com/users/guyboertje/events{/privacy}","received_events_url":"https://api.github.com/users/guyboertje/received_events","type":"User","site_admin":false},"labels":[{"id":227219964,"url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/labels/needs%20reviewing","name":"needs reviewing","color":"cc317c","default":false}],"state":"open","locked":false,"assignee":{"login":"guyboertje","id":84912,"avatar_url":"https://avatars2.githubusercontent.com/u/84912?v=4","gravatar_id":"","url":"https://api.github.com/users/guyboertje","html_url":"https://github.com/guyboertje","followers_url":"https://api.github.com/users/guyboertje/followers","following_url":"https://api.github.com/users/guyboertje/following{/other_user}","gists_url":"https://api.github.com/users/guyboertje/gists{/gist_id}","starred_url":"https://api.github.com/users/guyboertje/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guyboertje/subscriptions","organizations_url":"https://api.github.com/users/guyboertje/orgs","repos_url":"https://api.github.com/users/guyboertje/repos","events_url":"https://api.github.com/users/guyboertje/events{/privacy}","received_events_url":"https://api.github.com/users/guyboertje/received_events","type":"User","site_admin":false},"assignees":[{"login":"guyboertje","id":84912,"avatar_url":"https://avatars2.githubusercontent.com/u/84912?v=4","gravatar_id":"","url":"https://api.github.com/users/guyboertje","html_url":"https://github.com/guyboertje","followers_url":"https://api.github.com/users/guyboertje/followers","following_url":"https://api.github.com/users/guyboertje/following{/other_user}","gists_url":"https://api.github.com/users/guyboertje/gists{/gist_id}","starred_url":"https://api.github.com/users/guyboertje/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guyboertje/subscriptions","organizations_url":"https://api.github.com/users/guyboertje/orgs","repos_url":"https://api.github.com/users/guyboertje/repos","events_url":"https://api.github.com/users/guyboertje/events{/privacy}","received_events_url":"https://api.github.com/users/guyboertje/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2018-05-24T13:36:53Z","updated_at":"2018-05-30T15:54:56Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/pulls/192","html_url":"https://github.com/logstash-plugins/logstash-input-file/pull/192","diff_url":"https://github.com/logstash-plugins/logstash-input-file/pull/192.diff","patch_url":"https://github.com/logstash-plugins/logstash-input-file/pull/192.patch"},"body":"TIL The difference in Ruby between `IO#stat` and `IO.stat`, former takes a stat on the io (file) object and the latter takes a stat on a path string (as does `File::Stat.new` and Pathname#stat).\r\n\r\nWhen an open file is rotated the watched_file `path` may be pointing at a different inode with new content, and if so, the opened file is still pointing to the original inode and content. We have used `IO.stat` or `File::Stat.new` (since 2012) on the path regardless of whether the file is open or closed - this is not ideal.\r\n\r\nThis PR adds the ability to do path based stat on closed files and path plus io based stat on open files. By incrementing `pending_inode_count` when we detect inode changes between the two kinds of stat calls and we also cache the previous size of the old content and the new content.\r\nWhen the inode is different between the two stat structures we know that the file is rotated. When we detect this we can fully read the old content as we still have an open file handle. At the end of the full read we close the file io and reset the watched_file and sincedb.\r\n**NOTE**: we do detect inode changes for watched_files that are not `active` (i.e. closed, ignored, watched) but we can't know the renamed path of the old content in order to read any content we may not have processed while the watched_file was not `active`. A WARN message is logged in this case. In practice this only likely to occur on not `active` files when tailing more than 4095 (or `max_open_files` if set) files AND the file is rotated before active files can be put into the `closed` state - files become eligible for close when the last read was more than`close_older` ago AND the file is fully read to the last stat size.\r\nHandle race conditions better:\r\n1) `stat` is taken bang in the middle of the rotation rename cascade,  `f -> f.1` | `stat` | `tmp -> f`\r\n2) Discover finds the renamed path before the delete reaction can be done (this only occurs when the renamed path is inside the `path` glob pattern- rare).\r\n\r\nMost of this code has been tested by a discuss user who had problems with rotations.\r\n\r\nFixes #188 "},"comment":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/comments/393215586","html_url":"https://github.com/logstash-plugins/logstash-input-file/pull/192#issuecomment-393215586","issue_url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/192","id":393215586,"user":{"login":"colinsurprenant","id":2010,"avatar_url":"https://avatars1.githubusercontent.com/u/2010?v=4","gravatar_id":"","url":"https://api.github.com/users/colinsurprenant","html_url":"https://github.com/colinsurprenant","followers_url":"https://api.github.com/users/colinsurprenant/followers","following_url":"https://api.github.com/users/colinsurprenant/following{/other_user}","gists_url":"https://api.github.com/users/colinsurprenant/gists{/gist_id}","starred_url":"https://api.github.com/users/colinsurprenant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colinsurprenant/subscriptions","organizations_url":"https://api.github.com/users/colinsurprenant/orgs","repos_url":"https://api.github.com/users/colinsurprenant/repos","events_url":"https://api.github.com/users/colinsurprenant/events{/privacy}","received_events_url":"https://api.github.com/users/colinsurprenant/received_events","type":"User","site_admin":false},"created_at":"2018-05-30T15:54:56Z","updated_at":"2018-05-30T15:54:56Z","author_association":"CONTRIBUTOR","body":"@guyboertje ah damn ... I did not see your last comment before starting. oh well. "}}
{"actor":{"display_login":"colinsurprenant"}}