{"action":"closed","issue":{"url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/208","repository_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch","labels_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/208/labels{/name}","comments_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/208/comments","events_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/208/events","html_url":"https://github.com/uken/fluent-plugin-elasticsearch/issues/208","id":185581223,"number":208,"title":"ES Transport forgets username and password [401]","user":{"login":"se-m","id":642156,"avatar_url":"https://avatars.githubusercontent.com/u/642156?v=3","gravatar_id":"","url":"https://api.github.com/users/se-m","html_url":"https://github.com/se-m","followers_url":"https://api.github.com/users/se-m/followers","following_url":"https://api.github.com/users/se-m/following{/other_user}","gists_url":"https://api.github.com/users/se-m/gists{/gist_id}","starred_url":"https://api.github.com/users/se-m/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/se-m/subscriptions","organizations_url":"https://api.github.com/users/se-m/orgs","repos_url":"https://api.github.com/users/se-m/repos","events_url":"https://api.github.com/users/se-m/events{/privacy}","received_events_url":"https://api.github.com/users/se-m/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-27T06:10:03Z","updated_at":"2016-10-27T17:50:28Z","closed_at":"2016-10-27T17:50:28Z","body":"elasticsearch:2.4 + shield\r\ngem 'fluent-plugin-elasticsearch' version '1.7.0'\r\ngem 'fluent-plugin-parser' version '0.6.1'\r\ngem 'fluent-plugin-record-reformer' version '0.8.2'\r\ngem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'\r\ngem 'fluentd' version '0.12.24'\r\n\r\nElastic and fluentd in dockers\r\n\r\nI don't know how is this possible, but i faced with this behavior.\r\nAll sevices are set and all working. But periodically fluentd stops send data to es at all. Fluentd configuration reload solves the problem as long as it does not happen again. In the beginning, I did not notice this problem. Period was 4-5 days when i noticed it. But I get this error every day now.  The PC is not very powerfull and when fluentd failing to flush it's ok, but 401 error is beyond my understanding.\r\n\r\nfluentd  log:\r\n```\r\n2016-10-26 14:20:01 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-10-26 14:20:02 +0000 \r\n            error_class=\"Elasticsearch::Transport::Transport::Errors::Unauthorized\" error=\"[401] \r\n            {\\\"error\\\":\r\n              {\\\"root_cause\\\":[\r\n                {\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request [/_bulk]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"shield\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}\r\n              ]\r\n             ,\\\"type\\\":\\\"security_exception\\\"\r\n             ,\\\"reason\\\":\\\"missing authentication token for REST request [/_bulk]\\\"\r\n             ,\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"shield\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}\r\n             ,\\\"status\\\":401}\" plugin_id=\"object:2ac356475ca4\"\r\n             \r\n  2016-10-26 14:20:01 +0000 [warn]: /home/fluent/.gem/ruby/2.2.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:201:in `__raise_transport_error'\r\n  2016-10-26 14:20:01 +0000 [warn]: /home/fluent/.gem/ruby/2.2.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:312:in `perform_request'\r\n  2016-10-26 14:20:01 +0000 [warn]: /home/fluent/.gem/ruby/2.2.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'\r\n  2016-10-26 14:20:01 +0000 [warn]: /home/fluent/.gem/ruby/2.2.0/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in `perform_request'\r\n  2016-10-26 14:20:01 +0000 [warn]: /home/fluent/.gem/ruby/2.2.0/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/bulk.rb:90:in `bulk'\r\n  2016-10-26 14:20:01 +0000 [warn]: /home/fluent/.gem/ruby/2.2.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/out_elasticsearch.rb:334:in `send'\r\n  2016-10-26 14:20:01 +0000 [warn]: /home/fluent/.gem/ruby/2.2.0/gems/fluent-plugin-elasticsearch-1.7.0/lib/fluent/plugin/out_elasticsearch.rb:318:in `write'\r\n  2016-10-26 14:20:01 +0000 [warn]: /usr/lib/ruby/gems/2.2.0/gems/fluentd-0.12.24/lib/fluent/buffer.rb:354:in `write_chunk'\r\n  2016-10-26 14:20:01 +0000 [warn]: /usr/lib/ruby/gems/2.2.0/gems/fluentd-0.12.24/lib/fluent/buffer.rb:333:in `pop'\r\n  2016-10-26 14:20:01 +0000 [warn]: /usr/lib/ruby/gems/2.2.0/gems/fluentd-0.12.24/lib/fluent/output.rb:338:in `try_flush'\r\n  2016-10-26 14:20:01 +0000 [warn]: /usr/lib/ruby/gems/2.2.0/gems/fluentd-0.12.24/lib/fluent/output.rb:149:in `run'\r\n  2016-10-26 14:20:01 +0000 fluent.warn: {\"next_retry\":\"2016-10-26 14:20:02 +0000\",\"error_class\":\"Elasticsearch::Transport::Transport::Errors::Unauthorized\",\"error\":\"[401] {\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request [/_bulk]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"shield\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request [/_bulk]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"shield\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"plugin_id\":\"object:2ac356475ca4\",\"message\":\"temporarily failed to flush the buffer. next_retry=2016-10-26 14:20:02 +0000 error_class=\\\"Elasticsearch::Transport::Transport::Errors::Unauthorized\\\" error=\\\"[401] {\\\\\\\"error\\\\\\\":{\\\\\\\"root_cause\\\\\\\":[{\\\\\\\"type\\\\\\\":\\\\\\\"security_exception\\\\\\\",\\\\\\\"reason\\\\\\\":\\\\\\\"missing authentication token for REST request [/_bulk]\\\\\\\",\\\\\\\"header\\\\\\\":{\\\\\\\"WWW-Authenticate\\\\\\\":\\\\\\\"Basic realm=\\\\\\\\\\\\\\\"shield\\\\\\\\\\\\\\\" charset=\\\\\\\\\\\\\\\"UTF-8\\\\\\\\\\\\\\\"\\\\\\\"}}],\\\\\\\"type\\\\\\\":\\\\\\\"security_exception\\\\\\\",\\\\\\\"reason\\\\\\\":\\\\\\\"missing authentication token for REST request [/_bulk]\\\\\\\",\\\\\\\"header\\\\\\\":{\\\\\\\"WWW-Authenticate\\\\\\\":\\\\\\\"Basic realm=\\\\\\\\\\\\\\\"shield\\\\\\\\\\\\\\\" charset=\\\\\\\\\\\\\\\"UTF-8\\\\\\\\\\\\\\\"\\\\\\\"}},\\\\\\\"status\\\\\\\":401}\\\" plugin_id=\\\"object:2ac356475ca4\\\"\"}\r\n  2016-10-26 14:20:02 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-10-26 14:20:04 +0000 error_class=\"Elasticsearch::Transport::Transport::Errors::Unauthorized\" error=\"[401] {\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request [/_bulk]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"shield\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request [/_bulk]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"shield\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\" plugin_id=\"object:2ac356475ca4\"\r\n  2016-10-26 14:20:02 +0000 [warn]: suppressed same stacktrace\r\n  2016-10-26 14:20:02 +0000 fluent.warn: {\"next_retry\":\"2016-10-26 14:20:04 +0000\",\"error_class\":\"Elasticsearch::Transport::Transport::Errors::Unauthorized\",\"error\":\"[401] {\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request [/_bulk]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"shield\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication token for REST request [/_bulk]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"shield\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"plugin_id\":\"object:2ac356475ca4\",\"message\":\"temporarily failed to flush the buffer. next_retry=2016-10-26 14:20:04 +0000 error_class=\\\"Elasticsearch::Transport::Transport::Errors::Unauthorized\\\" error=\\\"[401] {\\\\\\\"error\\\\\\\":{\\\\\\\"root_cause\\\\\\\":[{\\\\\\\"type\\\\\\\":\\\\\\\"security_exception\\\\\\\",\\\\\\\"reason\\\\\\\":\\\\\\\"missing authentication token for REST request [/_bulk]\\\\\\\",\\\\\\\"header\\\\\\\":{\\\\\\\"WWW-Authenticate\\\\\\\":\\\\\\\"Basic realm=\\\\\\\\\\\\\\\"shield\\\\\\\\\\\\\\\" charset=\\\\\\\\\\\\\\\"UTF-8\\\\\\\\\\\\\\\"\\\\\\\"}}],\\\\\\\"type\\\\\\\":\\\\\\\"security_exception\\\\\\\",\\\\\\\"reason\\\\\\\":\\\\\\\"missing authentication token for REST request [/_bulk]\\\\\\\",\\\\\\\"header\\\\\\\":{\\\\\\\"WWW-Authenticate\\\\\\\":\\\\\\\"Basic realm=\\\\\\\\\\\\\\\"shield\\\\\\\\\\\\\\\" charset=\\\\\\\\\\\\\\\"UTF-8\\\\\\\\\\\\\\\"\\\\\\\"}},\\\\\\\"status\\\\\\\":401}\\\" plugin_id=\\\"object:2ac356475ca4\\\"\"}\r\n```\r\n\r\n"}}
{"actor":{"display_login":"pitr"}}