{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/kibana/issues/21204","repository_url":"https://api.github.com/repos/elastic/kibana","labels_url":"https://api.github.com/repos/elastic/kibana/issues/21204/labels{/name}","comments_url":"https://api.github.com/repos/elastic/kibana/issues/21204/comments","events_url":"https://api.github.com/repos/elastic/kibana/issues/21204/events","html_url":"https://github.com/elastic/kibana/issues/21204","id":344425440,"node_id":"MDU6SXNzdWUzNDQ0MjU0NDA=","number":21204,"title":"[Infra UI] Reconstruct log message from parsed data","user":{"login":"weltenwort","id":973741,"node_id":"MDQ6VXNlcjk3Mzc0MQ==","avatar_url":"https://avatars3.githubusercontent.com/u/973741?v=4","gravatar_id":"","url":"https://api.github.com/users/weltenwort","html_url":"https://github.com/weltenwort","followers_url":"https://api.github.com/users/weltenwort/followers","following_url":"https://api.github.com/users/weltenwort/following{/other_user}","gists_url":"https://api.github.com/users/weltenwort/gists{/gist_id}","starred_url":"https://api.github.com/users/weltenwort/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/weltenwort/subscriptions","organizations_url":"https://api.github.com/users/weltenwort/orgs","repos_url":"https://api.github.com/users/weltenwort/repos","events_url":"https://api.github.com/users/weltenwort/events{/privacy}","received_events_url":"https://api.github.com/users/weltenwort/received_events","type":"User","site_admin":false},"labels":[{"id":983061515,"node_id":"MDU6TGFiZWw5ODMwNjE1MTU=","url":"https://api.github.com/repos/elastic/kibana/labels/:Infra%20UI","name":":Infra UI","color":"6cddd6","default":false},{"id":981817023,"node_id":"MDU6TGFiZWw5ODE4MTcwMjM=","url":"https://api.github.com/repos/elastic/kibana/labels/:Ingest%20UI","name":":Ingest UI","color":"c5def5","default":false},{"id":1019680159,"node_id":"MDU6TGFiZWwxMDE5NjgwMTU5","url":"https://api.github.com/repos/elastic/kibana/labels/loe:multi-week","name":"loe:multi-week","color":"DB1374","default":false}],"state":"closed","locked":false,"assignee":{"login":"weltenwort","id":973741,"node_id":"MDQ6VXNlcjk3Mzc0MQ==","avatar_url":"https://avatars3.githubusercontent.com/u/973741?v=4","gravatar_id":"","url":"https://api.github.com/users/weltenwort","html_url":"https://github.com/weltenwort","followers_url":"https://api.github.com/users/weltenwort/followers","following_url":"https://api.github.com/users/weltenwort/following{/other_user}","gists_url":"https://api.github.com/users/weltenwort/gists{/gist_id}","starred_url":"https://api.github.com/users/weltenwort/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/weltenwort/subscriptions","organizations_url":"https://api.github.com/users/weltenwort/orgs","repos_url":"https://api.github.com/users/weltenwort/repos","events_url":"https://api.github.com/users/weltenwort/events{/privacy}","received_events_url":"https://api.github.com/users/weltenwort/received_events","type":"User","site_admin":false},"assignees":[{"login":"weltenwort","id":973741,"node_id":"MDQ6VXNlcjk3Mzc0MQ==","avatar_url":"https://avatars3.githubusercontent.com/u/973741?v=4","gravatar_id":"","url":"https://api.github.com/users/weltenwort","html_url":"https://github.com/weltenwort","followers_url":"https://api.github.com/users/weltenwort/followers","following_url":"https://api.github.com/users/weltenwort/following{/other_user}","gists_url":"https://api.github.com/users/weltenwort/gists{/gist_id}","starred_url":"https://api.github.com/users/weltenwort/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/weltenwort/subscriptions","organizations_url":"https://api.github.com/users/weltenwort/orgs","repos_url":"https://api.github.com/users/weltenwort/repos","events_url":"https://api.github.com/users/weltenwort/events{/privacy}","received_events_url":"https://api.github.com/users/weltenwort/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-07-25T12:37:25Z","updated_at":"2018-09-18T09:55:43Z","closed_at":"2018-09-18T09:55:43Z","author_association":"CONTRIBUTOR","body":"This issues outlines the way in which the log messages displayed in the logging \r\nui part of the Infra UI are derived from the documents stored in Elasticsearch.\r\n\r\n# Task Breakdown\r\n\r\n* [x] Implement graphql query to request log entries from a source (with a naive, static rule list) (PR #21306)\r\n* [x] Update logging ui to consume the new graphql query (PR #21706)\r\n* [ ] Add formatting rules for filebeat modules\r\n* [ ] Add generic, robust fallback formatting rules\r\n* [ ] Add configuration options to allow the user to specify custom rules\r\n\r\n# Rationale\r\n\r\nSimply displaying \"the log message\" as-is from a specific field in the \r\ndocuments retrieved from Elasticsearch is not practical for several reasons:\r\n\r\n* **lack of a message**: Documents produced by some beats modules (and other\r\n  sources) do not always have a single \"message\" field. Reasons might be that\r\n  the message has been forwarded in a parsed format by a third party or that\r\n  the message representation is highly use-case-dependent.\r\n* **normalization**: When aggregating logs from different sources in a\r\n  centralized store and UI the disparities between the log formats become\r\n  apparent. In order to improve the user experience when viewing log messages\r\n  from multiple sources we want to benefit from the extraction and\r\n  normalization that has in many cases already been performed as part of the\r\n  ingest pipeline. (e.g. timestamps, IP addresses)\r\n* **enrichment**: The document might already have been enriched with details \r\n  from additional sources during ingestion, that were not part of the original \r\n  message.\r\n* **interactivity**: Knowing the substructure of a synthesized log message \r\n  enables the UI components to improve the rendering and provide deeper \r\n  interactivity with parts of the message (e.g. add a filter by clicking on a \r\n  hostname).\r\n\r\n# Specification\r\n\r\nThe message reconstruction process is paramtereized by a set of predefined \r\nrules as well as rules provided by the user in the configuration (see \r\n\"Formatting Rules\" below).\r\n\r\nThe retrieval, transformation and rendering can be broken down into the \r\nfollowing steps:\r\n\r\n1. Querying\r\n   1. Load list of formatting rules\r\n   2. Extract set of required fields from formatting conditions and instructions\r\n   3. Fetch documents including the required fields\r\n2. Extraction\r\n   1. Apply the formatting rules to each document with the first match winning\r\n   2. Transmit the response to the browser including a sequence of tokens for \r\n      each entry which were resolved as described in the formatting instructions\r\n3. Rendering\r\n   1. Render the tokens as specified in the response for each log entry\r\n\r\n## Formatting Rules\r\n\r\nThe rules are kept in an ordered list that encodes the relative priority of \r\neach rule. A rule is composed of a formatting condition and formatting \r\ninstructions in a declarative manner. The declarative characteristic allows for \r\nderivation of the set of required document fields, which is useful for \r\nvalidation and optimization.\r\n\r\n```typescript\r\ninterface FormattingRule {\r\n  condition: FormattingCondition;\r\n  format: FormattingInstructions;\r\n}\r\n```\r\n\r\nThe **formatting condition** is used to decide whether to use the corresponding \r\nformatting instruction to format the log event. It is either a list of \r\nfield-value-pairs that the event document needs to contain in order to match or \r\na list of fields that need to be present in order to match.\r\n\r\n```typescript\r\ntype FormattingCondition =\r\n  | FormattingFieldValueCondition\r\n  | FormattingFieldExistsCondition;\r\n\r\ninterface FormattingFieldValueCondition {\r\n  values: {\r\n    [fieldName: string]: any;\r\n  };\r\n}\r\n\r\ninterface FormattingFieldExistsCondition {\r\n  exists: string[];\r\n}\r\n```\r\n\r\nThe **formatting instructions** declare how the log message can be derived from \r\nthe document fields. This is done by defining a list of tokens that either \r\nrepresent a field from the documents or a string constant. The tokens will be \r\nconcatenated to produce the message.\r\n\r\n```typescript\r\ntype FormattingInstructions = FormattingInstruction[];\r\n\r\ntype FormattingInstruction =\r\n  | FormattingFieldReference\r\n  | FormattingConstant\r\n\r\ninterface FormattingFieldReference {\r\n  field: string;\r\n}\r\n\r\ninterface FormattingConstant {\r\n  constant: string;\r\n}\r\n```\r\n\r\n# Considerations\r\n\r\n## Where should the synthesis happen?\r\n\r\nThe synthesis could be performed in three different locations of the data \r\nretrieval pipeline:\r\n\r\n* **In the query**: A painless scripted field could construct the message from \r\n  various document fields. While that would minimize the amount of data that \r\n  needs to be transferred from Elasticsearch to Kibana, it would deny knowledge \r\n  about the message substructe to the client. Additionally, programmatically \r\n  generating the painless scripts increases the complexity while reducing \r\n  testability.\r\n* **In the Kibana server**: The Kibana server could retrieve the documents and \r\n  synthesize the message for delivery to the client. That would keep all the \r\n  processing logic in TypeScript and minimize the amount of data transferred to \r\n  the browser. The UI components would still lack deeper knowledge about the \r\n  message substructure.\r\n* **In the browser**: Performing the reconstruction in the browser would yield \r\n  ultimate rendering flexibility at the cost of an increased data transfer \r\n  volume.\r\n\r\nFlexibility and ease of implementation suggests the third option would be preferable."},"comment":{"url":"https://api.github.com/repos/elastic/kibana/issues/comments/422331355","html_url":"https://github.com/elastic/kibana/issues/21204#issuecomment-422331355","issue_url":"https://api.github.com/repos/elastic/kibana/issues/21204","id":422331355,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMjMzMTM1NQ==","user":{"login":"weltenwort","id":973741,"node_id":"MDQ6VXNlcjk3Mzc0MQ==","avatar_url":"https://avatars3.githubusercontent.com/u/973741?v=4","gravatar_id":"","url":"https://api.github.com/users/weltenwort","html_url":"https://github.com/weltenwort","followers_url":"https://api.github.com/users/weltenwort/followers","following_url":"https://api.github.com/users/weltenwort/following{/other_user}","gists_url":"https://api.github.com/users/weltenwort/gists{/gist_id}","starred_url":"https://api.github.com/users/weltenwort/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/weltenwort/subscriptions","organizations_url":"https://api.github.com/users/weltenwort/orgs","repos_url":"https://api.github.com/users/weltenwort/repos","events_url":"https://api.github.com/users/weltenwort/events{/privacy}","received_events_url":"https://api.github.com/users/weltenwort/received_events","type":"User","site_admin":false},"created_at":"2018-09-18T09:55:43Z","updated_at":"2018-09-18T09:55:43Z","author_association":"CONTRIBUTOR","body":"Closing this to avoid confusion about the progress. For reconstruction rules will evolve over time. Individual PRs will be created as needed."}}
{"actor":{"display_login":"weltenwort"}}