{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/4401","labels_url":"https://api.github.com/repos/elastic/logstash/issues/4401/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/4401/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/4401/events","html_url":"https://github.com/elastic/logstash/issues/4401","id":124104415,"number":4401,"title":"Use a settings file to complement command line argument parsing","user":{"login":"jsvd","id":31809,"avatar_url":"https://avatars.githubusercontent.com/u/31809?v=3","gravatar_id":"","url":"https://api.github.com/users/jsvd","html_url":"https://github.com/jsvd","followers_url":"https://api.github.com/users/jsvd/followers","following_url":"https://api.github.com/users/jsvd/following{/other_user}","gists_url":"https://api.github.com/users/jsvd/gists{/gist_id}","starred_url":"https://api.github.com/users/jsvd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsvd/subscriptions","organizations_url":"https://api.github.com/users/jsvd/orgs","repos_url":"https://api.github.com/users/jsvd/repos","events_url":"https://api.github.com/users/jsvd/events{/privacy}","received_events_url":"https://api.github.com/users/jsvd/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/logstash/labels/config%20management","name":"config management","color":"bfd4f2"},{"url":"https://api.github.com/repos/elastic/logstash/labels/design","name":"design","color":"eb6420"},{"url":"https://api.github.com/repos/elastic/logstash/labels/discuss","name":"discuss","color":"cc317c"},{"url":"https://api.github.com/repos/elastic/logstash/labels/manageability","name":"manageability","color":"c7def8"},{"url":"https://api.github.com/repos/elastic/logstash/labels/v3.0.0","name":"v3.0.0","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-12-28T18:31:40Z","updated_at":"2016-01-18T20:13:42Z","closed_at":null,"body":"Currently logstash lists the following command line arguments:\r\n\r\n```\r\n    -f, --config CONFIG_PATH      Load the logstash config from a specific file\r\n                                  or directory.  If a directory is given, all\r\n                                  files in that directory will be concatenated\r\n                                  in lexicographical order and then parsed as a\r\n                                  single config file. You can also specify\r\n                                  wildcards (globs) and any matched files will\r\n                                  be loaded in the order described above.\r\n    -e CONFIG_STRING              Use the given string as the configuration\r\n                                  data. Same syntax as the config file. If no\r\n                                  input is specified, then the following is\r\n                                  used as the default input:\r\n                                  \"input { stdin { type => stdin } }\"\r\n                                  and if no output is specified, then the\r\n                                  following is used as the default output:\r\n                                  \"output { stdout { codec => rubydebug } }\"\r\n                                  If you wish to use both defaults, please use\r\n                                  the empty string for the '-e' flag.\r\n                                   (default: \"\")\r\n    -w, --filterworkers COUNT     Sets the number of filter workers to run.\r\n                                   (default: 0)\r\n    -l, --log FILE                Write logstash internal logs to the given\r\n                                  file. Without this flag, logstash will emit\r\n                                  logs to standard output.\r\n    -v                            Increase verbosity of logstash internal logs.\r\n                                  Specifying once will show 'informational'\r\n                                  logs. Specifying twice will show 'debug'\r\n                                  logs. This flag is deprecated. You should use\r\n                                  --verbose or --debug instead.\r\n    --quiet                       Quieter logstash logging. This causes only \r\n                                  errors to be emitted.\r\n    --verbose                     More verbose logging. This causes 'info' \r\n                                  level logs to be emitted.\r\n    --debug                       Most verbose logging. This causes 'debug'\r\n                                  level logs to be emitted.\r\n    -V, --version                 Emit the version of logstash and its friends,\r\n                                  then exit.\r\n    -p, --pluginpath PATH         A path of where to find plugins. This flag\r\n                                  can be given multiple times to include\r\n                                  multiple paths. Plugins are expected to be\r\n                                  in a specific directory hierarchy:\r\n                                  'PATH/logstash/TYPE/NAME.rb' where TYPE is\r\n                                  'inputs' 'filters', 'outputs' or 'codecs'\r\n                                  and NAME is the name of the plugin.\r\n    -t, --configtest              Check configuration for valid syntax and then exit.\r\n    --[no-]allow-unsafe-shutdown  Force logstash to exit during shutdown even\r\n                                  if there are still inflight events in memory.\r\n                                  By default, logstash will refuse to quit until all\r\n                                  received events have been pushed to the outputs.\r\n                                   (default: false)\r\n    -h, --help                    print help\r\n```\r\n\r\nWith the introduction of the next generation pipeline and node identification these flags will be added:\r\n\r\n```\r\n    -w, --pipeline-workers COUNT  Sets the number of pipeline workers to run.\r\n                                   (default: 2)\r\n    -b, --pipeline-batch-size SIZE Size of batches the pipeline is to work in.\r\n                                   (default: 125)\r\n    -u, --pipeline-batch-delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling\r\n                                  for the next event.\r\n                                   (default: 5)\r\n```\r\n\r\nWith the pluggable agents, metrics, automatic reloading and more, the big amount of setting flags will clog the cli so I believe it's the time to introduce a settings file.\r\n\r\n### Implementation\r\n\r\nThe settings file should work together with the command line argument parser so that, for example, the user can either specify --node-name in the CLI or \"node.name\" in the settings file to set the name of the node.\r\n\r\n#### Option 1: adding a settings section to the config file\r\n\r\nPros:\r\n  + a single place to check for both the pipeline config and settings\r\n  + consistent syntax between pipeline config and settings\r\n\r\nCons:\r\n  - many people use multiple config files, how to support multiple `config {}` blocks?\r\n  - existing treetop code and AST is painful to manage and change\r\n  - with automatic config reloading and potentially remote loading it's useful to ensure the two concepts are kept separately\r\n\r\n#### Option 2: individual YAML file for settings\r\n\r\nPros:\r\n  - easy to understand hierarchical structure to configure core components: pipeline, agent, logger\r\n  - better settings experience across elasticsearch, kibana and beats\r\n\r\nCons:\r\n  - It's YAML\r\n\r\n### Common Implementation Problems:\r\n\r\n  - No cli arg parsing tool accepts a settings file so the integration between cli args and settings file must be written by us\r\n\r\n\r\n### Example of YAML file:\r\n\r\n```yaml\r\nnode:\r\n  name: myindexer\r\npipeline:\r\n  workers: 2\r\n  batch-size: 125\r\n  batch-delay: 5\r\n  allow-unsafe-shutdown: false\r\nlogging:\r\n  verbose: false\r\n  debug: false\r\n  file:\r\ncore:\r\n  pluginpath: # $LOGSTASH_HOME/plugins\r\n```\r\n\r\n### What now?\r\n\r\nWhat do you think about this? Comments appreciated!"},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/172641966","html_url":"https://github.com/elastic/logstash/issues/4401#issuecomment-172641966","issue_url":"https://api.github.com/repos/elastic/logstash/issues/4401","id":172641966,"user":{"login":"suyograo","id":1595958,"avatar_url":"https://avatars.githubusercontent.com/u/1595958?v=3","gravatar_id":"","url":"https://api.github.com/users/suyograo","html_url":"https://github.com/suyograo","followers_url":"https://api.github.com/users/suyograo/followers","following_url":"https://api.github.com/users/suyograo/following{/other_user}","gists_url":"https://api.github.com/users/suyograo/gists{/gist_id}","starred_url":"https://api.github.com/users/suyograo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suyograo/subscriptions","organizations_url":"https://api.github.com/users/suyograo/orgs","repos_url":"https://api.github.com/users/suyograo/repos","events_url":"https://api.github.com/users/suyograo/events{/privacy}","received_events_url":"https://api.github.com/users/suyograo/received_events","type":"User","site_admin":false},"created_at":"2016-01-18T20:13:42Z","updated_at":"2016-01-18T20:13:42Z","body":"Many users may already have a `LS_HOME/config` directory"}}