{"action":"created","issue":{"url":"https://api.github.com/repos/fluent/fluentd/issues/1218","repository_url":"https://api.github.com/repos/fluent/fluentd","labels_url":"https://api.github.com/repos/fluent/fluentd/issues/1218/labels{/name}","comments_url":"https://api.github.com/repos/fluent/fluentd/issues/1218/comments","events_url":"https://api.github.com/repos/fluent/fluentd/issues/1218/events","html_url":"https://github.com/fluent/fluentd/issues/1218","id":175820374,"number":1218,"title":"Failed to write data into buffer by buffer overflow","user":{"login":"stachdude","id":1833861,"avatar_url":"https://avatars.githubusercontent.com/u/1833861?v=3","gravatar_id":"","url":"https://api.github.com/users/stachdude","html_url":"https://github.com/stachdude","followers_url":"https://api.github.com/users/stachdude/followers","following_url":"https://api.github.com/users/stachdude/following{/other_user}","gists_url":"https://api.github.com/users/stachdude/gists{/gist_id}","starred_url":"https://api.github.com/users/stachdude/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stachdude/subscriptions","organizations_url":"https://api.github.com/users/stachdude/orgs","repos_url":"https://api.github.com/users/stachdude/repos","events_url":"https://api.github.com/users/stachdude/events{/privacy}","received_events_url":"https://api.github.com/users/stachdude/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-09-08T17:58:11Z","updated_at":"2016-09-08T18:11:03Z","closed_at":null,"body":"- fluentd 0.14.4 - 0.14.6 \r\n- Centos 7\r\n- Your configuration\r\n\r\n```\r\n<source>\r\n    @type tail\r\n    format /REGEX_HERE/\r\n    tag http.*\r\n    path /data/data.log\r\n</source>\r\n\r\n\r\n<match **>\r\n  @type forward\r\n  send_timeout 60s\r\n  recover_wait 10s\r\n  heartbeat_interval 1s\r\n  phi_threshold 16\r\n  hard_timeout 60s\r\n\r\n  <server>\r\n    host 172.16.1.1\r\n    port 24224\r\n    weight 60\r\n  </server>\r\n</match>\r\n```\r\n\r\nAfter a couple o minutes of log tail (logs are streaming) I get this warn/error \r\n\r\n```\r\n  2016-09-08 17:41:16 +0000 [warn]: suppressed same stacktrace\r\n2016-09-08 17:41:16 +0000 [error]: failed to emit fluentd's log event tag=\"fluent.warn\" event={\"message\"=>\"failed to write data into buffer by buffer overflow\"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\"\r\n2016-09-08 17:41:16 +0000 [warn]: failed to write data into buffer by buffer overflow\r\n2016-09-08 17:41:16 +0000 [warn]: emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" tag=\"fluent.warn\"\r\n  2016-09-08 17:41:16 +0000 [warn]: suppressed same stacktrace\r\n2016-09-08 17:41:16 +0000 [error]: failed to emit fluentd's log event tag=\"fluent.warn\" event={\"error\"=>\"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>\", \"tag\"=>\"<tag_here>\", \"message\"=>\"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\\\"buffer space has too many data\\\" tag=\\\"<tag_here>\\\"\"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\"\r\n2016-09-08 17:41:16 +0000 [warn]: failed to write data into buffer by buffer overflow\r\n2016-09-08 17:41:16 +0000 [warn]: emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" tag=\"<tag_here>\"\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/buffer.rb:186:in `write'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:704:in `block in handle_stream_simple'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:610:in `write_guard'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:703:in `handle_stream_simple'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:600:in `execute_chunking'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:544:in `emit_buffered'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/event_router.rb:90:in `emit_stream'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/in_tail.rb:310:in `receive_lines'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/in_tail.rb:420:in `wrap_receive_lines'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/in_tail.rb:579:in `on_notify'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/in_tail.rb:444:in `on_notify'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin_helper/timer.rb:76:in `on_timer'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run_once'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin_helper/event_loop.rb:71:in `block in start'\r\n  2016-09-08 17:41:16 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'\r\n2016-09-08 17:41:17 +0000 [warn]: failed to write data into buffer by buffer overflow\r\n2016-09-08 17:41:17 +0000 [warn]: emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" tag=\"fluent.warn\"\r\n  2016-09-08 17:41:17 +0000 [warn]: suppressed same stacktrace\r\n2016-09-08 17:41:17 +0000 [error]: failed to emit fluentd's log event tag=\"fluent.warn\" event={\"message\"=>\"failed to write data into buffer by buffer overflow\"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\"\r\n2016-09-08 17:41:17 +0000 [warn]: failed to write data into buffer by buffer overflow\r\n2016-09-08 17:41:17 +0000 [warn]: emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" tag=\"fluent.warn\"\r\n  2016-09-08 17:41:17 +0000 [warn]: suppressed same stacktrace\r\n2016-09-08 17:41:17 +0000 [warn]: failed to write data into buffer by buffer overflow\r\n2016-09-08 17:41:17 +0000 [warn]: emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" tag=\"<tag_here>\"\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/buffer.rb:186:in `write'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:704:in `block in handle_stream_simple'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:610:in `write_guard'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:703:in `handle_stream_simple'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:600:in `execute_chunking'\r\n2016-09-08 17:41:17 +0000 [error]: failed to emit fluentd's log event tag=\"fluent.warn\" event={\"error\"=>\"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>\", \"tag\"=>\"<tag_here>\", \"message\"=>\"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\\\"buffer space has too many data\\\" tag=\\\"<tag_here>\\\"\"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\"\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:544:in `emit_buffered'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/event_router.rb:90:in `emit_stream'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/in_tail.rb:310:in `receive_lines'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/in_tail.rb:420:in `wrap_receive_lines'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/in_tail.rb:579:in `on_notify'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/in_tail.rb:444:in `on_notify'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/in_tail.rb:529:in `on_change'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run_once'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin_helper/event_loop.rb:71:in `block in start'\r\n  2016-09-08 17:41:17 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'\r\n2016-09-08 17:41:17 +0000 [warn]: failed to write data into buffer by buffer overflow\r\n2016-09-08 17:41:17 +0000 [warn]: emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" tag=\"<tag_here>\"\r\n```\r\n\r\nFrom that moment on, logs are filling up with those error/warns.\r\n\r\nRegex is extracting around 36 uniq fields, and the stream flow is around 2-3k eps.\r\n\r\nThe flow is not stopped, and the process is running, however at this stage I'm unable to ascertain is there any data lost due to this behavior."},"comment":{"url":"https://api.github.com/repos/fluent/fluentd/issues/comments/245687746","html_url":"https://github.com/fluent/fluentd/issues/1218#issuecomment-245687746","issue_url":"https://api.github.com/repos/fluent/fluentd/issues/1218","id":245687746,"user":{"login":"repeatedly","id":16928,"avatar_url":"https://avatars.githubusercontent.com/u/16928?v=3","gravatar_id":"","url":"https://api.github.com/users/repeatedly","html_url":"https://github.com/repeatedly","followers_url":"https://api.github.com/users/repeatedly/followers","following_url":"https://api.github.com/users/repeatedly/following{/other_user}","gists_url":"https://api.github.com/users/repeatedly/gists{/gist_id}","starred_url":"https://api.github.com/users/repeatedly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/repeatedly/subscriptions","organizations_url":"https://api.github.com/users/repeatedly/orgs","repos_url":"https://api.github.com/users/repeatedly/repos","events_url":"https://api.github.com/users/repeatedly/events{/privacy}","received_events_url":"https://api.github.com/users/repeatedly/received_events","type":"User","site_admin":false},"created_at":"2016-09-08T18:11:03Z","updated_at":"2016-09-08T18:11:03Z","body":"> Regex is extracting around 36 uniq fields, and the stream flow is around 2-3k eps.\r\n\r\nIt seems normal traffic.\r\nDoes this happen with fluentd v0.12?\r\nIf so, your aggregator doesn't have a lack of capacity.\r\nIf this happens with only fluentd v0.14, we need to check buffer flush implementation."}}
{"actor":{"display_login":"repeatedly"}}