{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/6366","repository_url":"https://api.github.com/repos/elastic/logstash","labels_url":"https://api.github.com/repos/elastic/logstash/issues/6366/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/6366/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/6366/events","html_url":"https://github.com/elastic/logstash/issues/6366","id":193921178,"node_id":"MDU6SXNzdWUxOTM5MjExNzg=","number":6366,"title":"Logstash 2.4.1/5.0.x: Environments Variables parsed as Lists/Arrays for Logstash Configuration","user":{"login":"berglh","id":12455338,"node_id":"MDQ6VXNlcjEyNDU1MzM4","avatar_url":"https://avatars1.githubusercontent.com/u/12455338?v=4","gravatar_id":"","url":"https://api.github.com/users/berglh","html_url":"https://github.com/berglh","followers_url":"https://api.github.com/users/berglh/followers","following_url":"https://api.github.com/users/berglh/following{/other_user}","gists_url":"https://api.github.com/users/berglh/gists{/gist_id}","starred_url":"https://api.github.com/users/berglh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/berglh/subscriptions","organizations_url":"https://api.github.com/users/berglh/orgs","repos_url":"https://api.github.com/users/berglh/repos","events_url":"https://api.github.com/users/berglh/events{/privacy}","received_events_url":"https://api.github.com/users/berglh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":21,"created_at":"2016-12-07T00:00:34Z","updated_at":"2018-08-22T22:57:45Z","closed_at":null,"author_association":"NONE","body":"I've been replacing the use of `logstash-filter-environment` plugin with the new feature: [Using Environment Variables in the Configuration](https://www.elastic.co/guide/en/logstash/current/environment-variables.html#_setting_the_tcp_port). \r\n\r\nI initially incorrectly diagnosed and reported the issue here [ES hosts array: Support for parsing Environment Variables](https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/520#issuecomment-265085666). @jordansissel advised I should file an issue in the correct project for this particular feature.\r\n\r\nOne of the things I am attempting to do is to supply an array for configuration item such as the hosts array in the `logstash-output-elasticsearch` plugin. This is useful when the environment hostnames and IPs are ephemeral and the use of a service discovery via query of `etcd` to determine IP and ports of online elasticsearch nodes during logstash startup is useful.\r\n\r\nI currently achieve this using sed to replace a placeholder string in the config on the docker container startup. It's not a major issue, but considering environment parsing functionality is built into Logstash now, it'd be great to leverage this instead of hacking the config prior to launch.\r\n\r\n- Version: 2.4.1 & 5.0.2\r\n- Operating System: Oracle Enterprise Linux\r\n- Docker: 1.12.2\r\n- Sample Data:\r\n  - With a host string:\r\n```\r\n~/logstash-5.0.2$ ES_HOSTS=\"localhost:9200\" bin/logstash -e 'output { elasticsearch { hosts => \"${ES_HOSTS}\" } }'\r\n\r\nSending Logstash's logs to /home/somebody/logstash-5.0.2/logs which is now configured via log4j2.properties\r\nThe stdin plugin is now waiting for input:\r\n[2016-12-06T16:07:20,738][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[\"http://localhost:9200\"]}}\r\n[2016-12-06T16:07:20,740][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}\r\n[2016-12-06T16:07:20,781][WARN ][logstash.outputs.elasticsearch] Marking url as dead. {:reason=>\"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection refused (Connection refused)\", :url=>#<URI::HTTP:0x41764dd8 URL:http://localhost:9200>, :error_message=>\"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection refused (Connection refused)\", :error_class=>\"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError\"}\r\n```\r\n- Sample Data\r\n  - With a host array:\r\n```\r\nuser@computer:~/logstash-5.0.2$ export ES_HOSTS=\"\\\"localhost:9200\\\", \\\"host2:9200\\\", \\\"host3:9200\\\"\"\r\n\r\nuser@computer:~/logstash-5.0.2$ echo $ES_HOSTS\r\n\"localhost:9200\", \"host2:9200\", \"host3:9200\"\r\n\r\nuser@computer:~/logstash-5.0.2$ bin/logstash -e 'output { elasticsearch { hosts => [ \"${ES_HOSTS}\" ] } }'\r\nSending Logstash's logs to ~/logstash-5.0.2/logs which is now configured via log4j2.properties\r\nThe stdin plugin is now waiting for input:\r\n[2016-12-06T16:17:02,070][ERROR][logstash.agent           ] Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: Host '\"localhost:9200\", \"host2:9200\", \"host3:9200\"' was specified, but is not valid! Use either a full URL or a hostname:port string!>, :backtrace=>[\"~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:183:in `host_to_url'\", \"~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:93:in `build_pool'\", \"org/jruby/RubyArray.java:2414:in `map'\", \"~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:93:in `build_pool'\", \"~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:20:in `initialize'\", \"~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:53:in `build'\", \"~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch.rb:188:in `build_client'\", \"~/logstash-5.0.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.2.1-java/lib/logstash/outputs/elasticsearch/common.rb:13:in `register'\", \"~/logstash-5.0.2/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:8:in `register'\", \"~/logstash-5.0.2/logstash-core/lib/logstash/output_delegator.rb:37:in `register'\", \"~/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:196:in `start_workers'\", \"org/jruby/RubyArray.java:1613:in `each'\", \"~/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:196:in `start_workers'\", \"~/logstash-5.0.2/logstash-core/lib/logstash/pipeline.rb:153:in `run'\", \"~/logstash-5.0.2/logstash-core/lib/logstash/agent.rb:250:in `start_pipeline'\"]}\r\n[2016-12-06T16:17:02,086][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}\r\n[2016-12-06T16:17:05,074][WARN ][logstash.agent           ] stopping pipeline {:id=>\"main\"}\r\n\r\nuser@computer:~/logstash-5.0.2$ bin/logstash -e 'output { elasticsearch { hosts => [ \"localhost:9200\", \"host2:9200\", \"host3:9200\" ] } }'\r\nSending Logstash's logs to ~/logstash-5.0.2/logs which is now configured via log4j2.properties\r\nThe stdin plugin is now waiting for input:\r\n[2016-12-06T16:17:21,387][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[\"http://localhost:9200\", \"http://host2:9200\", \"http://host3:9200\"]}}\r\n[2016-12-06T16:17:21,389][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}\r\n[2016-12-06T16:17:21,444][WARN ][logstash.outputs.elasticsearch] Marking url as dead. {:reason=>\"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection refused (Connection refused)\", :url=>#<URI::HTTP:0x51dd94f4 URL:http://localhost:9200>, :error_message=>\"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection refused (Connection refused)\", :error_class=>\"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError\"}\r\n[2016-12-06T16:17:21,445][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>\"Elasticsearch Unreachable: [http://localhost:9200][Manticore::SocketException] Connection refused (Connection refused)\", :class=>\"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError\"}\r\n[2016-12-06T16:17:21,446][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>\"LogStash::Outputs::ElasticSearch\", :hosts=>[\"localhost:9200\", \"host2:9200\", \"host3:9200\"]}\r\n[2016-12-06T16:17:21,447][INFO ][logstash.pipeline        ] Starting pipeline {\"id\"=>\"main\", \"pipeline.workers\"=>8, \"pipeline.batch.size\"=>125, \"pipeline.batch.delay\"=>5, \"pipeline.max_inflight\"=>1000}\r\n[2016-12-06T16:17:21,449][INFO ][logstash.pipeline        ] Pipeline main started\r\n[2016-12-06T16:17:21,467][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}\r\n[2016-12-06T16:17:26,390][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:url=>#<URI::HTTP:0x588852e4 URL:http://localhost:9200>, :healthcheck_path=>\"/\"}\r\n^C[2016-12-06T16:17:29,147][WARN ][logstash.runner          ] SIGINT received. Shutting down the agent.\r\n[2016-12-06T16:17:29,151][WARN ][logstash.agent           ] stopping pipeline {:id=>\"main\"}\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/415213299","html_url":"https://github.com/elastic/logstash/issues/6366#issuecomment-415213299","issue_url":"https://api.github.com/repos/elastic/logstash/issues/6366","id":415213299,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNTIxMzI5OQ==","user":{"login":"fbaligand","id":3750300,"node_id":"MDQ6VXNlcjM3NTAzMDA=","avatar_url":"https://avatars1.githubusercontent.com/u/3750300?v=4","gravatar_id":"","url":"https://api.github.com/users/fbaligand","html_url":"https://github.com/fbaligand","followers_url":"https://api.github.com/users/fbaligand/followers","following_url":"https://api.github.com/users/fbaligand/following{/other_user}","gists_url":"https://api.github.com/users/fbaligand/gists{/gist_id}","starred_url":"https://api.github.com/users/fbaligand/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fbaligand/subscriptions","organizations_url":"https://api.github.com/users/fbaligand/orgs","repos_url":"https://api.github.com/users/fbaligand/repos","events_url":"https://api.github.com/users/fbaligand/events{/privacy}","received_events_url":"https://api.github.com/users/fbaligand/received_events","type":"User","site_admin":false},"created_at":"2018-08-22T22:57:45Z","updated_at":"2018-08-22T22:57:45Z","author_association":"CONTRIBUTOR","body":"Hi @jordansissel ,\r\n\r\nTo answer to your question about values that contain comma, I think that : \r\n- first, IMHO, this is really a rare case for array environment variable injection : concerning date format for example, most of the time, it is not specific to an environment. \r\n- then, I searched over spring documentation (which manages comma separated values for env var injection) and didn't find anything about how to handle values that contain comma. It seems to be not managed, although this feature is existing and used for years by lots of people. So it seems not to be a big problem for spring user community.\r\n\r\nThat's just my humble opinion :)"}}
{"actor":{"display_login":"fbaligand"}}