{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/kibana/issues/19286","repository_url":"https://api.github.com/repos/elastic/kibana","labels_url":"https://api.github.com/repos/elastic/kibana/issues/19286/labels{/name}","comments_url":"https://api.github.com/repos/elastic/kibana/issues/19286/comments","events_url":"https://api.github.com/repos/elastic/kibana/issues/19286/events","html_url":"https://github.com/elastic/kibana/issues/19286","id":325089652,"number":19286,"title":"[Logstash Management] Pipeline Builder","user":{"login":"ycombinator","id":51061,"avatar_url":"https://avatars2.githubusercontent.com/u/51061?v=4","gravatar_id":"","url":"https://api.github.com/users/ycombinator","html_url":"https://github.com/ycombinator","followers_url":"https://api.github.com/users/ycombinator/followers","following_url":"https://api.github.com/users/ycombinator/following{/other_user}","gists_url":"https://api.github.com/users/ycombinator/gists{/gist_id}","starred_url":"https://api.github.com/users/ycombinator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ycombinator/subscriptions","organizations_url":"https://api.github.com/users/ycombinator/orgs","repos_url":"https://api.github.com/users/ycombinator/repos","events_url":"https://api.github.com/users/ycombinator/events{/privacy}","received_events_url":"https://api.github.com/users/ycombinator/received_events","type":"User","site_admin":false},"labels":[{"id":832655738,"url":"https://api.github.com/repos/elastic/kibana/labels/:Logstash","name":":Logstash","color":"bfdadc","default":false},{"id":470241819,"url":"https://api.github.com/repos/elastic/kibana/labels/:Management","name":":Management","color":"fef2c0","default":false},{"id":477020958,"url":"https://api.github.com/repos/elastic/kibana/labels/Meta","name":"Meta","color":"e11d21","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-21T23:18:38Z","updated_at":"2018-05-22T20:32:06Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"## Motivation\r\nAKA: What problem would a Logstash Pipeline Builder solve?\r\n\r\n### Broaden the user base\r\nToday, users are able to centrally manage Logstash pipelines through the Kibana UI. However, they need to type in the Logstash pipeline configuration in a textarea, as shown below.\r\n\r\n![image3](https://user-images.githubusercontent.com/51061/40334090-ae2e5ad0-5d10-11e8-920c-41ac53bf6074.png)\r\n\r\nThis UX works for users who are familiar with the Logstash syntax.\r\n\r\nHowever, there are users who are either not as familiar with the Logstash syntax or are generally more comfortable using graphical interfaces featuring drag-n-drop, etc. over textual interfaces. The Pipeline Builder UI will allow such users to easily create Logstash pipelines.\r\n\r\n### Better discoverability of plugins and their settings\r\n\r\nWhen using the textarea interface to type in the pipeline configuration, users need to know which plugins they want to use and what settings these plugins take. This creates a discoverability problem for users. While we can alleviate this problem with auto-complete, that can only go so far within the confines of a textual editor.\r\n\r\nIn comparison, the Pipeline Builder UI can provide a richer discoverability experience. Particularly, it can show users searchable lists of plugins, with short descriptions, allowing them to better narrow down the plugin they are looking for, as shown below.\r\n\r\n![image1](https://user-images.githubusercontent.com/51061/40334108-c1d7eeca-5d10-11e8-8649-d50543509ffa.png)\r\n\r\nSimilarly, when it comes time to populate a plugin's settings, the UI can show them a form for the settings with appropriate data type constraints and validation on the form fields.\r\n\r\n![image2](https://user-images.githubusercontent.com/51061/40334115-cbc7d67a-5d10-11e8-9814-d38287dc286a.png)\r\n\r\nIn both cases (plugin discovery and settings discovery), the UI can provide contextual links to complete documentation on our web site.\r\n\r\n### Express data flows with greater flexibility\r\nToday, certain data flows are cumbersome to express in the Logstash pipeline configuration syntax. For example, say you want to send events to Elasticsearch or S3 depending on some property of the data, and only want to perform geoip enrichment for events going into Elasticsearch. The pipeline would be expressed as follows:\r\n\r\n\r\n```\r\ninput {\r\n  …\r\n}\r\n\r\nfilter {\r\n  … \r\n\r\n  if [condition] {\r\n    geoip { … }\r\n  }\r\n}\r\n\r\noutput {\r\n  if [condition] {\r\n    elasticsearch { … }\r\n  } else {\r\n    s3 { … }\r\n}\r\n```\r\n\r\nNotice the repeated if condition in the filter section and the output section.\r\n\r\nThis sort of repetition can be avoided with a Pipeline Builder UI, as shown below.\r\n\r\n![image4](https://user-images.githubusercontent.com/51061/40334128-df623054-5d10-11e8-9559-910c68c8eee2.png)\r\n\r\nIn general, the Pipeline Builder UI can allow users to focus more on their _data flow_ and less on the _syntax_ for building a pipeline.\r\n\r\n## Implementation steps\r\n\r\n* [ ] **Create the Pipeline Builder GUI itself.** Much of this can be done using standard EUI components like `EuiPanel`, `EuiFlyout`, `EuiFlexGroup`, `EuiForm`, etc. Additionally, edges/arrows can be created using an SVG element that spans the entire \"drawable\" area.\r\n   * Generating the plugin list: For starters, this could be a \"hardcoded\" JSON \"manifest\" file that lives within the UI codebase. In the future this list could be dynamically populated at run time by running instances of Logstash.\r\n   * Settings for each plugin. For starters, when users edit a plugin's settings, they can be shown a rich text area where they can enter the plugin's settings in Logstash configuration syntax or YAML or JSON. In the future, these settings can be contained in the same dynamic registry as used by the plugin list.\r\n* [ ] **Make Logstash understand Pipeline Builder-created pipelines.** The Pipeline Builder will generate a JSON representation of a pipeline that matches very closely the LIR data structure used internally by Logstash today for executing pipelines.\r\n   * [ ] Logstash core will need to be enhanced to receive this JSON representation and deserialize it into the internal LIR data structure.\r\n   * [ ] One more subtlety that needs to be addressed for MVP is that around the syntax users should use to define their if conditions in the Pipeline Builder UI. In order for users to use the same syntax as they would today in a textual Logstash pipeline config, some refactoring needs to be done in core Logstash code first: the part of the grammar that is responsible for parsing if expressions will need to be ported over to Java so it could be called from LIR code (which is Java) as well as the current imperative syntax parsing code (which is JRuby).\r\n* [ ] **Make Monitoring Pipeline Viewer render Pipeline Builder-created pipelines.** Today Logstash processes ship pipeline representations into Monitoring. Each pipeline representation is a serialized representations of the constituent LIR objects. Logstash can continue to ship these pipeline representations as before. The only difference from today is that Pipeline Builder pipelines would contain metadata about relative vertex positions. The Pipeline Viewer UI in Monitoring can then use this position information to lay out the pipeline pretty much exactly as the user laid it out when creating it in the Pipeline Builder UI.\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/kibana/issues/comments/391130785","html_url":"https://github.com/elastic/kibana/issues/19286#issuecomment-391130785","issue_url":"https://api.github.com/repos/elastic/kibana/issues/19286","id":391130785,"user":{"login":"ycombinator","id":51061,"avatar_url":"https://avatars2.githubusercontent.com/u/51061?v=4","gravatar_id":"","url":"https://api.github.com/users/ycombinator","html_url":"https://github.com/ycombinator","followers_url":"https://api.github.com/users/ycombinator/followers","following_url":"https://api.github.com/users/ycombinator/following{/other_user}","gists_url":"https://api.github.com/users/ycombinator/gists{/gist_id}","starred_url":"https://api.github.com/users/ycombinator/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ycombinator/subscriptions","organizations_url":"https://api.github.com/users/ycombinator/orgs","repos_url":"https://api.github.com/users/ycombinator/repos","events_url":"https://api.github.com/users/ycombinator/events{/privacy}","received_events_url":"https://api.github.com/users/ycombinator/received_events","type":"User","site_admin":false},"created_at":"2018-05-22T20:32:06Z","updated_at":"2018-05-22T20:32:06Z","author_association":"CONTRIBUTOR","body":"Thanks for those questions, @brandonmensing.\r\n\r\n> Can users 'onboard' their current pipelines? \r\n\r\nTurning on Centralized Management on a Logstash node implies turning off other ways of configuration (local files, etc.). So a user could today onboard their local configs by first turning on central management on all their LS nodes, then copy-pasting the local config into the UI from one of the nodes. For convenience we can provide a CLI tool to perform the same operation so the user doesn't have to manually copy-paste.\r\n\r\nOnce we have a config in central management, when we introduce the Pipeline Builder, we can also introduce a button in the current central management UI to promote a centrally-managed textual config to the pipeline builder. The builder would do a \"best effort\" layout and users could then move vertices around to lay things out as they want + make any edits too. Once they hit save users would not be able to edit the same pipeline via the text form UI (since the internal representations are quite different).\r\n\r\n> Can a user take a Builder pipeline and 'leave' the builder tool behind to use config files instead?\r\n\r\nI think generally no, but depending on the pipeline configuration we might be able to do a \"machine-generated\" conversion to a textual config file. What's the use case for this?\r\n\r\n>  Will any of this work across multiple versions of LS?\r\n\r\nAt the moment I think it's safe to say yes. In the future it might depend on what changes we make to the LS textual config grammar and the pipeline builder internal representation. We would, of course, try to keep things backwards compatible as much as possible and provide an upgrade path when we couldn't.\r\n\r\n"}}
{"actor":{"display_login":"ycombinator"}}