{"action":"created","issue":{"url":"https://api.github.com/repos/fluent/fluent-plugin-kafka/issues/78","repository_url":"https://api.github.com/repos/fluent/fluent-plugin-kafka","labels_url":"https://api.github.com/repos/fluent/fluent-plugin-kafka/issues/78/labels{/name}","comments_url":"https://api.github.com/repos/fluent/fluent-plugin-kafka/issues/78/comments","events_url":"https://api.github.com/repos/fluent/fluent-plugin-kafka/issues/78/events","html_url":"https://github.com/fluent/fluent-plugin-kafka/issues/78","id":181215654,"number":78,"title":"Lost messages, strange buffer files after upgrading","user":{"login":"dtboctor","id":5854168,"avatar_url":"https://avatars.githubusercontent.com/u/5854168?v=3","gravatar_id":"","url":"https://api.github.com/users/dtboctor","html_url":"https://github.com/dtboctor","followers_url":"https://api.github.com/users/dtboctor/followers","following_url":"https://api.github.com/users/dtboctor/following{/other_user}","gists_url":"https://api.github.com/users/dtboctor/gists{/gist_id}","starred_url":"https://api.github.com/users/dtboctor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dtboctor/subscriptions","organizations_url":"https://api.github.com/users/dtboctor/orgs","repos_url":"https://api.github.com/users/dtboctor/repos","events_url":"https://api.github.com/users/dtboctor/events{/privacy}","received_events_url":"https://api.github.com/users/dtboctor/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-10-05T17:22:53Z","updated_at":"2016-10-11T05:02:43Z","closed_at":null,"body":"We recently bumped our version of the plugin to the latest on Sept 21.  \r\n\r\nLater on when trying to consolidate data we realized a lot of data from that day is missing, and when I go to our folders where we are storing the buffers for fluentd I see a bunch of strange \"..*.log\" files from that day that I can't explain\r\n```\r\nWed Oct 05 09:56:20 [6:0]$ ls -lah\r\ntotal 1.4M\r\ndrwxr-xr-x. 2 lumoslabs lumoslabs 4.0K Oct  5 09:56 .\r\ndrwxr-xr-x. 4 root      root        41 Jul 14 16:42 ..\r\n-rw-r--r--. 1 root      root       93K Sep 21 14:01 ..b53d0adad2c926a4e.log\r\n-rw-r--r--. 1 root      root       78K Oct  5 09:56 buffer.b53e210e8a540d529c7409b2af3b263c8.log\r\n-rw-r--r--. 1 root      root        75 Oct  5 09:56 buffer.b53e210e8a540d529c7409b2af3b263c8.log.meta\r\n-rw-r--r--. 1 root      root      1.1M Oct  5 09:56 buffer.q53e210df2541167ab5cede2521dd8af6.log\r\n-rw-r--r--. 1 root      root        77 Oct  5 09:56 buffer.q53e210df2541167ab5cede2521dd8af6.log.meta\r\n```\r\n\r\nDo these files contain our lost data?  (Specifically the `..b53d0adad2c926a4e.log` file).  Are they indicative that something broke, and can we get any data that might be in there out of there?   Is there a different procedure we should follow when bumping our version to prevent any chance of data loss? "},"comment":{"url":"https://api.github.com/repos/fluent/fluent-plugin-kafka/issues/comments/252814610","html_url":"https://github.com/fluent/fluent-plugin-kafka/issues/78#issuecomment-252814610","issue_url":"https://api.github.com/repos/fluent/fluent-plugin-kafka/issues/78","id":252814610,"user":{"login":"repeatedly","id":16928,"avatar_url":"https://avatars.githubusercontent.com/u/16928?v=3","gravatar_id":"","url":"https://api.github.com/users/repeatedly","html_url":"https://github.com/repeatedly","followers_url":"https://api.github.com/users/repeatedly/followers","following_url":"https://api.github.com/users/repeatedly/following{/other_user}","gists_url":"https://api.github.com/users/repeatedly/gists{/gist_id}","starred_url":"https://api.github.com/users/repeatedly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/repeatedly/subscriptions","organizations_url":"https://api.github.com/users/repeatedly/orgs","repos_url":"https://api.github.com/users/repeatedly/repos","events_url":"https://api.github.com/users/repeatedly/events{/privacy}","received_events_url":"https://api.github.com/users/repeatedly/received_events","type":"User","site_admin":false},"created_at":"2016-10-11T05:02:43Z","updated_at":"2016-10-11T05:02:43Z","body":"> Yeah, it looks like fluentd was bumped\r\n\r\nWe recommend to pin the fluentd / plugin version for stable deployment.\r\n\r\n> Is there any way to get those uploaded.\r\n\r\nOne way is moving old buffer files to other directly and using fluentd v0.12 to reload it.\r\nAnother way is reading buffer file and re-post it to fluentd via fluent-logger.\r\n\r\n> Was the new version of fluentd incompatible with the old buffer files?\r\n\r\nYes. fluentd v0.14 has new buffer mechanizm for more robustness and flexibility.\r\nHard to v0.12 buffers to v0.14 because v0.12 buffer doesn't have metadata in it."}}
{"actor":{"display_login":"repeatedly"}}