{"action":"created","issue":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/189","repository_url":"https://api.github.com/repos/logstash-plugins/logstash-input-file","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/189/labels{/name}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/189/comments","events_url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/189/events","html_url":"https://github.com/logstash-plugins/logstash-input-file/issues/189","id":322414156,"number":189,"title":"Zero byte read from NFS","user":{"login":"sv-gh","id":13316305,"avatar_url":"https://avatars0.githubusercontent.com/u/13316305?v=4","gravatar_id":"","url":"https://api.github.com/users/sv-gh","html_url":"https://github.com/sv-gh","followers_url":"https://api.github.com/users/sv-gh/followers","following_url":"https://api.github.com/users/sv-gh/following{/other_user}","gists_url":"https://api.github.com/users/sv-gh/gists{/gist_id}","starred_url":"https://api.github.com/users/sv-gh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sv-gh/subscriptions","organizations_url":"https://api.github.com/users/sv-gh/orgs","repos_url":"https://api.github.com/users/sv-gh/repos","events_url":"https://api.github.com/users/sv-gh/events{/privacy}","received_events_url":"https://api.github.com/users/sv-gh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-05-11T19:49:23Z","updated_at":"2018-05-14T21:38:46Z","closed_at":null,"author_association":"NONE","body":"logstash 6.2.4, logstash-input-file (4.0.5), Centos 6.4, NFS v3.2.29;\r\nConfig: input file, filter ruby, output file;\r\nSample - text log files;\r\nSteps to reproduce: NFS Server, several programs writing logs to NFS mounted directory, Logstash with 'input file' from several NFS mounted logs;\r\n\r\nWhen reading input files from NFS,\r\n     FileWatch::WatchedFile.file_read(amount)\r\nfunction uses @file.sysread(amount) to read data,\r\nthis sometime produce result buffer partially filled\r\nwith '\\0' bytes starting with some position.\r\n\r\nWe could reliably reproduce the problem on NFS v3.\r\n\r\nSuggested fix:\r\n- check zero byte in result, if present, wait, seek back, and re-read.\r\n\r\nRuby code fix in lib/filewatch/watched_file.rb :\r\n```\r\n    def file_read(amount)\r\n      #debug \"*** file_read #{path}\"\r\n      cc = 3\r\n      dt = 1.0/128\r\n      loop do\r\n        set_accessed_at\r\n        buf = @file.sysread(amount)\r\n        return buf unless zc = buf.index(\"\\0\")\r\n        amount = buf.bytesize # change amount to read\r\n        # warn \"*** ZERO-#{cc} byte (#{zc} of #{amount}) in #{path}\"\r\n        cc -= 1\r\n        return buf if cc.zero?\r\n        sleep(dt)\r\n        dt *= 4\r\n        @file.sysseek(-amount, IO::SEEK_CUR)\r\n      end\r\n    end \r\n```\r\nFixed, in filewatch gem, tested with NFS v.3, running OK under heavy NFS operations\r\nload (actively updated log files from numerous client boxes).\r\n"},"comment":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/comments/388971559","html_url":"https://github.com/logstash-plugins/logstash-input-file/issues/189#issuecomment-388971559","issue_url":"https://api.github.com/repos/logstash-plugins/logstash-input-file/issues/189","id":388971559,"user":{"login":"sv-gh","id":13316305,"avatar_url":"https://avatars0.githubusercontent.com/u/13316305?v=4","gravatar_id":"","url":"https://api.github.com/users/sv-gh","html_url":"https://github.com/sv-gh","followers_url":"https://api.github.com/users/sv-gh/followers","following_url":"https://api.github.com/users/sv-gh/following{/other_user}","gists_url":"https://api.github.com/users/sv-gh/gists{/gist_id}","starred_url":"https://api.github.com/users/sv-gh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sv-gh/subscriptions","organizations_url":"https://api.github.com/users/sv-gh/orgs","repos_url":"https://api.github.com/users/sv-gh/repos","events_url":"https://api.github.com/users/sv-gh/events{/privacy}","received_events_url":"https://api.github.com/users/sv-gh/received_events","type":"User","site_admin":false},"created_at":"2018-05-14T21:38:46Z","updated_at":"2018-05-14T21:38:46Z","author_association":"NONE","body":"\"It is not advised to use\" I translate to \"we don't know how to make it work, don't even try\", but lot of people did and not because they are masochistic, but for business needs.\r\n\r\n\"Big change and refactoring\" arguments are irrelevant in this discussion, sorry, please let me to ignore them.  You've completed great job and make lot of changes, I admit,\r\nbut they don't address/improve any issues reported in 'Read from NFS' cases. \r\n\r\n\"legitimate inclusion of NUL characters\" -- imho: NUL character in text log is quite some special case.\r\nContrary to this - \"Reading logs from NFS\" *IS* very valid use case, multiple issues raised before,\r\nshould be solved long time ago.\r\n\r\nIf you don't agree with suggested solution in general,\r\ncould you please introduce special option for\r\n\"Re-reading buffer with NUL char\"\r\nto provide at least partial support for poor Logstash users who must read logs from NFS v3.\r\n\r\n\"I will consider raising an exception\" - may I suggest to SEEK back and return empty string,\r\nthere is no need for global control transfer in processing well detected situation,\r\ncan we keep the code a little bit more clean and functional?"}}
{"actor":{"display_login":"sv-gh"}}