{"action":"created","issue":{"url":"https://api.github.com/repos/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/issues/116","repository_url":"https://api.github.com/repos/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs","labels_url":"https://api.github.com/repos/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/issues/116/labels{/name}","comments_url":"https://api.github.com/repos/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/issues/116/comments","events_url":"https://api.github.com/repos/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/issues/116/events","html_url":"https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/issues/116","id":335419474,"node_id":"MDU6SXNzdWUzMzU0MTk0NzQ=","number":116,"title":"unexpected error \"Member must have length greater than or equal to 1\"","user":{"login":"nunofernandes","id":1197165,"node_id":"MDQ6VXNlcjExOTcxNjU=","avatar_url":"https://avatars2.githubusercontent.com/u/1197165?v=4","gravatar_id":"","url":"https://api.github.com/users/nunofernandes","html_url":"https://github.com/nunofernandes","followers_url":"https://api.github.com/users/nunofernandes/followers","following_url":"https://api.github.com/users/nunofernandes/following{/other_user}","gists_url":"https://api.github.com/users/nunofernandes/gists{/gist_id}","starred_url":"https://api.github.com/users/nunofernandes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nunofernandes/subscriptions","organizations_url":"https://api.github.com/users/nunofernandes/orgs","repos_url":"https://api.github.com/users/nunofernandes/repos","events_url":"https://api.github.com/users/nunofernandes/events{/privacy}","received_events_url":"https://api.github.com/users/nunofernandes/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-06-25T13:58:30Z","updated_at":"2018-08-09T01:54:03Z","closed_at":null,"author_association":"NONE","body":"Hello,\r\n\r\nI'm using:\r\n```\r\n2018-06-25 13:47:31 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.5.0'\r\n2018-06-25 13:47:31 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.10.1'\r\n2018-06-25 13:47:31 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.4.0'\r\n2018-06-25 13:47:31 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.1.6'\r\n2018-06-25 13:47:31 +0000 [info]: gem 'fluent-plugin-kafka' version '0.6.5'\r\n2018-06-25 13:47:31 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.0.1'\r\n2018-06-25 13:47:31 +0000 [info]: gem 'fluent-plugin-s3' version '1.1.0'\r\n2018-06-25 13:47:31 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0'\r\n2018-06-25 13:47:31 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.3'\r\n2018-06-25 13:47:31 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.2'\r\n2018-06-25 13:47:31 +0000 [info]: gem 'fluentd' version '1.0.2'\r\n```\r\n\r\nAnd my fluentd is dying with:\r\n\r\n```\r\n2018-06-25 13:47:32 +0000 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::CloudwatchLogsInput title=:in_cloudwatch_logs_runner error_class=Aws::CloudWatchLogs::Errors::InvalidParameterException error=\"1 validation error detected: Value '' at 'nextToken' failed to satisfy constraint: Member must have length greater than or equal to 1\"\r\n2018-06-25 13:47:32 +0000 [error]: #0 unexpected error error_class=Aws::CloudWatchLogs::Errors::InvalidParameterException error=\"1 validation error detected: Value '' at 'nextToken' failed to satisfy constraint: Member must have length greater than or equal to 1\"\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.11.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.11.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.11.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.11.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.11.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.11.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.11.0/lib/seahorse/client/request.rb:70:in `send_request'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-cloudwatchlogs-1.3.0/lib/aws-sdk-cloudwatchlogs/client.rb:1154:in `get_log_events'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.5.0/lib/fluent/plugin/in_cloudwatch_logs.rb:146:in `get_events'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.5.0/lib/fluent/plugin/in_cloudwatch_logs.rb:112:in `block in run'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.5.0/lib/fluent/plugin/in_cloudwatch_logs.rb:110:in `each'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.5.0/lib/fluent/plugin/in_cloudwatch_logs.rb:110:in `run'\r\n  2018-06-25 13:47:32 +0000 [error]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.0.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'\r\n2018-06-25 13:47:32 +0000 [error]: #0 unexpected error error_class=Aws::CloudWatchLogs::Errors::InvalidParameterException error=\"1 validation error detected: Value '' at 'nextToken' failed to satisfy constraint: Member must have length greater than or equal to 1\"\r\n``` \r\n\r\nThe configuration is very simple:\r\n```\r\n<source>\r\n  @type cloudwatch_logs\r\n  region eu-west-1\r\n  tag devops.cloudwatch_logs.tst.systems\r\n  log_group_name logs-systems\r\n  log_stream_name logs\r\n  use_log_stream_name_prefix true\r\n  state_file /tmp/fluent-plugin-cloudwatch-tst-systems.state\r\n  format none\r\n</source>\r\n```\r\n\r\nIf I remove the state_file, it will fetch again all the logs but after any restart of the fluent service, it doesn't start again. Any ideas on how we can further debug this?"},"comment":{"url":"https://api.github.com/repos/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/issues/comments/411611758","html_url":"https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/issues/116#issuecomment-411611758","issue_url":"https://api.github.com/repos/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/issues/116","id":411611758,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTYxMTc1OA==","user":{"login":"barrytam20","id":16708979,"node_id":"MDQ6VXNlcjE2NzA4OTc5","avatar_url":"https://avatars2.githubusercontent.com/u/16708979?v=4","gravatar_id":"","url":"https://api.github.com/users/barrytam20","html_url":"https://github.com/barrytam20","followers_url":"https://api.github.com/users/barrytam20/followers","following_url":"https://api.github.com/users/barrytam20/following{/other_user}","gists_url":"https://api.github.com/users/barrytam20/gists{/gist_id}","starred_url":"https://api.github.com/users/barrytam20/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/barrytam20/subscriptions","organizations_url":"https://api.github.com/users/barrytam20/orgs","repos_url":"https://api.github.com/users/barrytam20/repos","events_url":"https://api.github.com/users/barrytam20/events{/privacy}","received_events_url":"https://api.github.com/users/barrytam20/received_events","type":"User","site_admin":false},"created_at":"2018-08-09T01:54:03Z","updated_at":"2018-08-09T01:54:03Z","author_association":"NONE","body":"for me, it works for a while then this error appears. Perhaps `store_next_token` is writing an empty string into the state file? "}}
{"actor":{"display_login":"barrytam20"}}