{"action":"created","issue":{"url":"https://api.github.com/repos/fluent/fluent-bit/issues/89","repository_url":"https://api.github.com/repos/fluent/fluent-bit","labels_url":"https://api.github.com/repos/fluent/fluent-bit/issues/89/labels{/name}","comments_url":"https://api.github.com/repos/fluent/fluent-bit/issues/89/comments","events_url":"https://api.github.com/repos/fluent/fluent-bit/issues/89/events","html_url":"https://github.com/fluent/fluent-bit/issues/89","id":167390648,"number":89,"title":"Buffer size of stdin?","user":{"login":"marktheunissen","id":453297,"avatar_url":"https://avatars.githubusercontent.com/u/453297?v=3","gravatar_id":"","url":"https://api.github.com/users/marktheunissen","html_url":"https://github.com/marktheunissen","followers_url":"https://api.github.com/users/marktheunissen/followers","following_url":"https://api.github.com/users/marktheunissen/following{/other_user}","gists_url":"https://api.github.com/users/marktheunissen/gists{/gist_id}","starred_url":"https://api.github.com/users/marktheunissen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marktheunissen/subscriptions","organizations_url":"https://api.github.com/users/marktheunissen/orgs","repos_url":"https://api.github.com/users/marktheunissen/repos","events_url":"https://api.github.com/users/marktheunissen/events{/privacy}","received_events_url":"https://api.github.com/users/marktheunissen/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"edsiper","id":369718,"avatar_url":"https://avatars.githubusercontent.com/u/369718?v=3","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false},"assignees":[{"login":"edsiper","id":369718,"avatar_url":"https://avatars.githubusercontent.com/u/369718?v=3","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false}],"milestone":null,"comments":13,"created_at":"2016-07-25T15:11:51Z","updated_at":"2016-07-27T13:50:39Z","closed_at":"2016-07-27T08:35:59Z","body":"Hi there, thanks for the cool utility. \r\n\r\nI'm trying to find out how big the buffer size is when reading from stdin. Let's say I have a 10M file and I just want to `cat` that file through fluent bit to my target.\r\n\r\nSo I do: `cat events.log | td-agent-bit -c td-agent-bit.conf` where the conf file is:\r\n\r\n```ini\r\n[SERVICE]\r\n    Flush     1\r\n    Daemon    off\r\n    Log_Level debug\r\n\r\n[INPUT]\r\n    Name stdin\r\n    Tag  events\r\n\r\n[OUTPUT]\r\n    Name forward\r\n    Match events\r\n    Host example.com\r\n    Port 24224\r\n```\r\n\r\nI would expect the entire 10M file to be sent through the pipeline, however the actual result is that I get:\r\n\r\n```\r\nTD-Agent Bit v0.8.3\r\nCopyright (C) Treasure Data\r\n\r\n[2016/07/25 16:27:12] [ info] starting engine\r\n[2016/07/25 16:27:12] [debug] [router] match rule stdin.0:forward.0\r\n[2016/07/25 16:27:14] [ warn] STDIN data incomplete, waiting for more data...\r\n```\r\n\r\nAnd only a tiny fraction of the file contents are delivered.\r\n\r\nNow, I can slow down the pipe using the following:\r\n\r\n`cat events.log | pv -l -L 10 -q | td-agent-bit -c td-agent-bit.conf`\r\n\r\nBut it still doesn't work as I'd expect, I get many more events delivered, but still just about 1/10th. And that's slowing the rate to 10 bytes / second.\r\n\r\nIs there a configuration toggle for the size of the buffer? I would like it to keep buffering data up to a certain limit, bounded by the amount of RAM on the box, I guess.\r\n\r\nThanks!"},"comment":{"url":"https://api.github.com/repos/fluent/fluent-bit/issues/comments/235590869","html_url":"https://github.com/fluent/fluent-bit/issues/89#issuecomment-235590869","issue_url":"https://api.github.com/repos/fluent/fluent-bit/issues/89","id":235590869,"user":{"login":"edsiper","id":369718,"avatar_url":"https://avatars.githubusercontent.com/u/369718?v=3","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false},"created_at":"2016-07-27T13:50:39Z","updated_at":"2016-07-27T13:50:39Z","body":"thanks, I will merge this new feature in 0.8 branch."}}
{"actor":{"display_login":"edsiper"}}