{"action":"created","issue":{"url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/112","repository_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch","labels_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/112/labels{/name}","comments_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/112/comments","events_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/112/events","html_url":"https://github.com/uken/fluent-plugin-elasticsearch/pull/112","id":88858606,"number":112,"title":"Invalid timestamps breaking plugin functionality","user":{"login":"gtrevg","id":1063149,"avatar_url":"https://avatars.githubusercontent.com/u/1063149?v=3","gravatar_id":"","url":"https://api.github.com/users/gtrevg","html_url":"https://github.com/gtrevg","followers_url":"https://api.github.com/users/gtrevg/followers","following_url":"https://api.github.com/users/gtrevg/following{/other_user}","gists_url":"https://api.github.com/users/gtrevg/gists{/gist_id}","starred_url":"https://api.github.com/users/gtrevg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gtrevg/subscriptions","organizations_url":"https://api.github.com/users/gtrevg/orgs","repos_url":"https://api.github.com/users/gtrevg/repos","events_url":"https://api.github.com/users/gtrevg/events{/privacy}","received_events_url":"https://api.github.com/users/gtrevg/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/labels/waiting-on-feedback","name":"waiting-on-feedback","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-06-16T22:39:11Z","updated_at":"2016-04-21T12:26:23Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/pulls/112","html_url":"https://github.com/uken/fluent-plugin-elasticsearch/pull/112","diff_url":"https://github.com/uken/fluent-plugin-elasticsearch/pull/112.diff","patch_url":"https://github.com/uken/fluent-plugin-elasticsearch/pull/112.patch"},"body":"If `logstash_format` is enabled and invalid timestamp data such as `nil` or a random string gets passed into the plugin via `@timestamp` or `time_key`, the plugin throws an exception and then gets stuck in a retry loop such as the following:\r\n\r\n```\r\n2015-06-15 17:01:22 -0700 [warn]: temporarily failed to flush the buffer. next_retry=2015-06-15 17:01:23 -0700 error_class=\"TypeError\" error=\"no implicit conversion of nil into String\" plugin_id=\"object:3f8d3da595b0\"\r\n  2015-06-15 17:01:22 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/time.rb:325:in `_parse'\r\n  2015-06-15 17:01:22 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/time.rb:325:in `parse'\r\n  2015-06-15 17:01:22 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-0.9.0/lib/fluent/plugin/out_elasticsearch.rb:131:in `block in write'\r\n  2015-06-15 17:01:22 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_memory.rb:61:in `feed_each'\r\n  2015-06-15 17:01:22 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_memory.rb:61:in `msgpack_each'\r\n  2015-06-15 17:01:22 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-0.9.0/lib/fluent/plugin/out_elasticsearch.rb:125:in `write'\r\n  2015-06-15 17:01:22 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:325:in `write_chunk'\r\n  2015-06-15 17:01:22 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:304:in `pop'\r\n  2015-06-15 17:01:22 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:321:in `try_flush'\r\n  2015-06-15 17:01:22 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:140:in `run'\r\n2015-06-15 17:01:24 -0700 [warn]: temporarily failed to flush the buffer. next_retry=2015-06-15 17:01:27 -0700 error_class=\"TypeError\" error=\"no implicit conversion of nil into String\" plugin_id=\"object:3f8d3da595b0\"\r\n  2015-06-15 17:01:24 -0700 [warn]: suppressed same stacktrace\r\n2015-06-15 17:01:28 -0700 [warn]: temporarily failed to flush the buffer. next_retry=2015-06-15 17:01:36 -0700 error_class=\"TypeError\" error=\"no implicit conversion of nil into String\" plugin_id=\"object:3f8d3da595b0\"\r\n  2015-06-15 17:01:28 -0700 [warn]: suppressed same stacktrace\r\n2015-06-15 17:01:37 -0700 [warn]: temporarily failed to flush the buffer. next_retry=2015-06-15 17:01:52 -0700 error_class=\"TypeError\" error=\"no implicit conversion of nil into String\" plugin_id=\"object:3f8d3da595b0\"\r\n```\r\n\r\nThe only way to fix the issue is to try to purge the bad data and/or restart the td-agent.\r\n\r\nThis pull request implements the solution of:\r\n\r\n1. Catching the exceptions:  `TypeError`, `ArgumentError`\r\n2. Uses the record time in place of the invalid time data\r\n3. logs a warning with the offending data\r\n\r\nTest cases that expose the issue and the solution are also included.\r\n\r\nThanks\r\n"},"comment":{"url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/comments/212893989","html_url":"https://github.com/uken/fluent-plugin-elasticsearch/pull/112#issuecomment-212893989","issue_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/112","id":212893989,"user":{"login":"pitr","id":102791,"avatar_url":"https://avatars.githubusercontent.com/u/102791?v=3","gravatar_id":"","url":"https://api.github.com/users/pitr","html_url":"https://github.com/pitr","followers_url":"https://api.github.com/users/pitr/followers","following_url":"https://api.github.com/users/pitr/following{/other_user}","gists_url":"https://api.github.com/users/pitr/gists{/gist_id}","starred_url":"https://api.github.com/users/pitr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pitr/subscriptions","organizations_url":"https://api.github.com/users/pitr/orgs","repos_url":"https://api.github.com/users/pitr/repos","events_url":"https://api.github.com/users/pitr/events{/privacy}","received_events_url":"https://api.github.com/users/pitr/received_events","type":"User","site_admin":false},"created_at":"2016-04-21T12:26:23Z","updated_at":"2016-04-21T12:26:23Z","body":"@MrMMorris I'll merge this as soon as all comments are addressed. If you would like to help, it would speed that up"}}