{"action":"created","issue":{"url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/441","repository_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch","labels_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/441/labels{/name}","comments_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/441/comments","events_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/441/events","html_url":"https://github.com/uken/fluent-plugin-elasticsearch/issues/441","id":337542553,"node_id":"MDU6SXNzdWUzMzc1NDI1NTM=","number":441,"title":"error log unreadable due to lots of backslahes","user":{"login":"amphibithen","id":9948555,"node_id":"MDQ6VXNlcjk5NDg1NTU=","avatar_url":"https://avatars1.githubusercontent.com/u/9948555?v=4","gravatar_id":"","url":"https://api.github.com/users/amphibithen","html_url":"https://github.com/amphibithen","followers_url":"https://api.github.com/users/amphibithen/followers","following_url":"https://api.github.com/users/amphibithen/following{/other_user}","gists_url":"https://api.github.com/users/amphibithen/gists{/gist_id}","starred_url":"https://api.github.com/users/amphibithen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amphibithen/subscriptions","organizations_url":"https://api.github.com/users/amphibithen/orgs","repos_url":"https://api.github.com/users/amphibithen/repos","events_url":"https://api.github.com/users/amphibithen/events{/privacy}","received_events_url":"https://api.github.com/users/amphibithen/received_events","type":"User","site_admin":false},"labels":[{"id":116194058,"node_id":"MDU6TGFiZWwxMTYxOTQwNTg=","url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/labels/waiting-on-feedback","name":"waiting-on-feedback","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-07-02T14:28:46Z","updated_at":"2018-07-24T15:47:45Z","closed_at":null,"author_association":"NONE","body":"#### Problem\r\nerror log is showing lots of backslashes between every field, making it near impossible to read through and lots of spam\r\n...\r\n\r\n#### Steps to replicate\r\ndockerfile\r\n```\r\nFROM fluent/fluentd:v1.2.2\r\n\r\nRUN apk add --update --virtual .build-deps \\\r\n        sudo build-base ruby-dev \\\r\n    && sudo gem install \\\r\n         fluent-plugin-elasticsearch \\\r\n         fluent-plugin-record-modifier \\\r\n    && sudo gem sources --clear-all \\\r\n    && apk del .build-deps \\\r\n    && rm -rf /var/cache/apk/* \\\r\n              /home/fluent/.gem/ruby/2.3.0/cache/*.gem\r\n\r\nENV ELASTICSEARCH_HOST=elasticsearch.mydomain.com \\\r\n    ELASTICSEARCH_PORT=443 \\\r\n    ELASTICSEARCH_PROTOCOL=https \\\r\n    cluster=test \\\r\n    vpc=test \\\r\n    host=test\r\n\r\nCOPY fluent.conf /fluentd/etc/\r\n```\r\n\r\nfluent.conf\r\n```\r\n<source>\r\n  @type forward\r\n  port 24224\r\n</source>\r\n\r\n<filter **/**:**>\r\n  @type record_modifier\r\n  <record>\r\n    host \"#{ENV['host']}\"\r\n    cluster \"#{ENV['cluster']}\"\r\n    vpc \"#{ENV['vpc']}\"\r\n    timestamp_ms ${ require 'time'; Time.now.utc.iso8601(3) }\r\n    container_id_trunc ${record[\"container_id\"][0..16]}\r\n    container_image ${tag}\r\n  </record>\r\n  <replace>\r\n    key container_image\r\n    expression /^?.+\\/(?<start>.+:)(?<end>.+)$/\r\n    replace \\k<start>\\k<end>\r\n  </replace>\r\n</filter>\r\n\r\n<match **>\r\n  @type copy\r\n  <store>\r\n    # reference https://github.com/uken/fluent-plugin-elasticsearch\r\n    @type elasticsearch\r\n    logstash_format true\r\n    time_key timestamp_ms\r\n    flush_interval 1s\r\n    host \"#{ENV['ELASTICSEARCH_HOST']}\"\r\n    port \"#{ENV['ELASTICSEARCH_PORT']}\"\r\n    scheme \"#{ENV['ELASTICSEARCH_PROTOCOL']}\"\r\n    ssl_verify false\r\n    ssl_version TLSv1_2\r\n    request_timeout 15s\r\n  </store>\r\n</match>\r\n```\r\n\r\nwhen an error shows up (typically i am getting ES rejected...no idea why but too hard to read through), error log shows (sorry i do not know where a new line starts or ends):\r\n```\r\nlocation=nil tag=\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"1234567890.dkr.ecr.us-east-1.amazonaws.com/mytestservice:20180622-1536_69\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" time=1529843902 record={\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"log\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"=>\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"6/24/18 12:38:22 PM UTC [INFO] [IntroscopeAgent.Properties] introscope.agent.metricAging.metricExclude.ignore.2=File System*\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\", \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"container_id\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"=>\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"707343b51e3774437d040781c8a81051308450b202efe451ac06401084636f5d\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\", \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"container_name\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"=>\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/ecs-mytestservice-30-mytestservice-ca9cb49da888ceb19e01\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\", \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"source\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"=>\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"stderr\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\", \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"host\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"=>\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"ip-10-149-5-134.bcbsnc.com\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\", \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"cluster\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"=>\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"general-shared\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\", \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"vpc\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"=>\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"vpc-abc1235\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\", \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"timestamp_ms\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"=>\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"2018-06-24T12:38:22.013Z\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\r\n```\r\n\r\n#### Expected Behavior or What you need to ask\r\n\r\nThere should be big blocks of `\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\` escaping each quote\r\n\r\n#### Using Fluentd and ES plugin versions\r\n\r\n* OS version  \r\n`v1.2/alpine/Dockerfile`\r\n\r\n* Fluentd v0.12 or v0.14/v1.0\r\n  * paste result of ``fluentd --version`` or ``td-agent --version``  \r\n`fluentd 1.2.2`\r\n\r\n* ES version (optional)\r\n`elasticsearch 6.2.4`\r\n\r\n* ES plugin 2.x.y or 1.x.y\r\n  * paste boot log of fluentd or td-agent\r\n  * paste result of ``fluent-gem list``, ``td-agent-gem list`` or your Gemfile.lock  \r\n```\r\n/ # fluent-gem list\r\n\r\n*** LOCAL GEMS ***\r\n\r\ncool.io (1.5.3)\r\ndig_rb (1.0.1)\r\nelasticsearch (6.1.0)\r\nelasticsearch-api (6.1.0)\r\nelasticsearch-transport (6.1.0)\r\nexcon (0.62.0)\r\nfaraday (0.15.2)\r\nfluent-plugin-elasticsearch (2.11.0)\r\nfluent-plugin-record-modifier (1.1.0)\r\nfluentd (1.2.2)\r\nhttp_parser.rb (0.6.0)\r\njson (2.1.0)\r\nmsgpack (1.2.4)\r\nmulti_json (1.13.1)\r\nmultipart-post (2.0.0)\r\noj (3.3.10)\r\nopenssl (default: 2.0.7)\r\npsych (default: 2.2.2)\r\nserverengine (2.0.6)\r\nsigdump (0.2.4)\r\nstrptime (0.2.3)\r\nthread_safe (0.3.6)\r\ntzinfo (1.2.5)\r\ntzinfo-data (1.2018.5)\r\nyajl-ruby (1.4.0)\r\n```\r\n\r\n```\r\n2018-07-02 13:45:03 +0000 [info]: parsing config file is succeeded path=\"/fluentd/etc/fluent.conf\"\r\n2018-07-02 13:45:03 +0000 [info]: 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour\r\n2018-07-02 13:45:03 +0000 [info]: Connection opened to Elasticsearch cluster => {:host=>\"elasticsearch.mydomain.com\", :port=>443, :scheme=>\"https\"}\r\n2018-07-02 13:45:03 +0000 [info]: Detected ES 6.x: ES 7.x will only accept `_doc` in type_name.\r\n2018-07-02 13:45:03 +0000 [info]: using configuration file: <ROOT>\r\n  <source>\r\n    @type forward\r\n    port 24224\r\n  </source>\r\n  <filter **/**:**>\r\n    @type record_modifier\r\n    <record>\r\n      host ip-10-149-4-26.bcbsnc.com\r\n      cluster general-shared\r\n      vpc vpc-abcd1234\r\n      timestamp_ms ${ require 'time'; Time.now.utc.iso8601(3) }\r\n      container_id_trunc ${record[\"container_id\"][0..16]}\r\n      container_image ${tag}\r\n    </record>\r\n    <replace>\r\n      key \"container_image\"\r\n      expression /^?.+\\/(?<start>.+:)(?<end>.+)$/\r\n      replace \"\\\\k<start>\\\\k<end>\"\r\n    </replace>\r\n  </filter>\r\n  <match **>\r\n    @type copy\r\n    <store>\r\n      @type \"elasticsearch\"\r\n      logstash_format true\r\n      time_key \"timestamp_ms\"\r\n      flush_interval 1s\r\n      host \"elasticsearch.mydomain.com\"\r\n      port 443\r\n      scheme \"https\"\r\n      ssl_verify false\r\n      ssl_version TLSv1_2\r\n      request_timeout 15s\r\n      <buffer>\r\n        flush_interval 1s\r\n      </buffer>\r\n    </store>\r\n  </match>\r\n</ROOT>\r\n2018-07-02 13:45:03 +0000 [info]: starting fluentd-1.2.2 pid=7 ruby=\"2.4.4\"\r\n2018-07-02 13:45:03 +0000 [info]: spawn command to main:  cmdline=[\"/usr/bin/ruby\", \"-Eascii-8bit:ascii-8bit\", \"/usr/bin/fluentd\", \"-c\", \"/fluentd/etc/fluent.conf\", \"-p\", \"/fluentd/plugins\", \"--under-supervisor\"]\r\n2018-07-02 13:45:04 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.11.0'\r\n2018-07-02 13:45:04 +0000 [info]: gem 'fluent-plugin-record-modifier' version '1.1.0'\r\n2018-07-02 13:45:04 +0000 [info]: gem 'fluentd' version '1.2.2'\r\n2018-07-02 13:45:04 +0000 [info]: adding filter pattern=\"**/**:**\" type=\"record_modifier\"\r\n2018-07-02 13:45:04 +0000 [info]: adding match pattern=\"**\" type=\"copy\"\r\n2018-07-02 13:45:04 +0000 [info]: #0 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour\r\n2018-07-02 13:45:04 +0000 [info]: #0 Connection opened to Elasticsearch cluster => {:host=>\"elasticsearch.mydomain.com\", :port=>443, :scheme=>\"https\"}\r\n2018-07-02 13:45:04 +0000 [info]: #0 Detected ES 6.x: ES 7.x will only accept `_doc` in type_name.\r\n2018-07-02 13:45:04 +0000 [info]: adding source type=\"forward\"\r\n2018-07-02 13:45:04 +0000 [info]: #0 starting fluentd worker pid=17 ppid=7 worker=0\r\n2018-07-02 13:45:04 +0000 [info]: #0 listening port port=24224 bind=\"0.0.0.0\"\r\n2018-07-02 13:45:04 +0000 [info]: #0 fluentd worker is now running worker=0\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/comments/407455111","html_url":"https://github.com/uken/fluent-plugin-elasticsearch/issues/441#issuecomment-407455111","issue_url":"https://api.github.com/repos/uken/fluent-plugin-elasticsearch/issues/441","id":407455111,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNzQ1NTExMQ==","user":{"login":"fernandosilvacornejo","id":579915,"node_id":"MDQ6VXNlcjU3OTkxNQ==","avatar_url":"https://avatars3.githubusercontent.com/u/579915?v=4","gravatar_id":"","url":"https://api.github.com/users/fernandosilvacornejo","html_url":"https://github.com/fernandosilvacornejo","followers_url":"https://api.github.com/users/fernandosilvacornejo/followers","following_url":"https://api.github.com/users/fernandosilvacornejo/following{/other_user}","gists_url":"https://api.github.com/users/fernandosilvacornejo/gists{/gist_id}","starred_url":"https://api.github.com/users/fernandosilvacornejo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fernandosilvacornejo/subscriptions","organizations_url":"https://api.github.com/users/fernandosilvacornejo/orgs","repos_url":"https://api.github.com/users/fernandosilvacornejo/repos","events_url":"https://api.github.com/users/fernandosilvacornejo/events{/privacy}","received_events_url":"https://api.github.com/users/fernandosilvacornejo/received_events","type":"User","site_admin":false},"created_at":"2018-07-24T15:47:45Z","updated_at":"2018-07-24T15:47:45Z","author_association":"NONE","body":"I can confirm those caleidoscopic backslash messages stop after blocking fluentd internal logs, as mentioned by @calinah."}}
{"actor":{"display_login":"fernandosilvacornejo"}}