{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/31046","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/31046/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/31046/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/31046/events","html_url":"https://github.com/elastic/elasticsearch/issues/31046","id":328742698,"node_id":"MDU6SXNzdWUzMjg3NDI2OTg=","number":31046,"title":"Audit logging structured entries","user":{"login":"albertzaharovits","id":4568420,"node_id":"MDQ6VXNlcjQ1Njg0MjA=","avatar_url":"https://avatars2.githubusercontent.com/u/4568420?v=4","gravatar_id":"","url":"https://api.github.com/users/albertzaharovits","html_url":"https://github.com/albertzaharovits","followers_url":"https://api.github.com/users/albertzaharovits/followers","following_url":"https://api.github.com/users/albertzaharovits/following{/other_user}","gists_url":"https://api.github.com/users/albertzaharovits/gists{/gist_id}","starred_url":"https://api.github.com/users/albertzaharovits/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/albertzaharovits/subscriptions","organizations_url":"https://api.github.com/users/albertzaharovits/orgs","repos_url":"https://api.github.com/users/albertzaharovits/repos","events_url":"https://api.github.com/users/albertzaharovits/events{/privacy}","received_events_url":"https://api.github.com/users/albertzaharovits/received_events","type":"User","site_admin":false},"labels":[{"id":912837734,"node_id":"MDU6TGFiZWw5MTI4Mzc3MzQ=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Security/Audit","name":":Security/Audit","color":"0e8a16","default":false},{"id":23172,"node_id":"MDU6TGFiZWwyMzE3Mg==","url":"https://api.github.com/repos/elastic/elasticsearch/labels/%3Efeature","name":">feature","color":"006b75","default":false},{"id":912911731,"node_id":"MDU6TGFiZWw5MTI5MTE3MzE=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/v6.4.0","name":"v6.4.0","color":"DDDDDD","default":false},{"id":484855464,"node_id":"MDU6TGFiZWw0ODQ4NTU0NjQ=","url":"https://api.github.com/repos/elastic/elasticsearch/labels/v7.0.0","name":"v7.0.0","color":"dddddd","default":false}],"state":"open","locked":false,"assignee":{"login":"albertzaharovits","id":4568420,"node_id":"MDQ6VXNlcjQ1Njg0MjA=","avatar_url":"https://avatars2.githubusercontent.com/u/4568420?v=4","gravatar_id":"","url":"https://api.github.com/users/albertzaharovits","html_url":"https://github.com/albertzaharovits","followers_url":"https://api.github.com/users/albertzaharovits/followers","following_url":"https://api.github.com/users/albertzaharovits/following{/other_user}","gists_url":"https://api.github.com/users/albertzaharovits/gists{/gist_id}","starred_url":"https://api.github.com/users/albertzaharovits/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/albertzaharovits/subscriptions","organizations_url":"https://api.github.com/users/albertzaharovits/orgs","repos_url":"https://api.github.com/users/albertzaharovits/repos","events_url":"https://api.github.com/users/albertzaharovits/events{/privacy}","received_events_url":"https://api.github.com/users/albertzaharovits/received_events","type":"User","site_admin":false},"assignees":[{"login":"albertzaharovits","id":4568420,"node_id":"MDQ6VXNlcjQ1Njg0MjA=","avatar_url":"https://avatars2.githubusercontent.com/u/4568420?v=4","gravatar_id":"","url":"https://api.github.com/users/albertzaharovits","html_url":"https://github.com/albertzaharovits","followers_url":"https://api.github.com/users/albertzaharovits/followers","following_url":"https://api.github.com/users/albertzaharovits/following{/other_user}","gists_url":"https://api.github.com/users/albertzaharovits/gists{/gist_id}","starred_url":"https://api.github.com/users/albertzaharovits/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/albertzaharovits/subscriptions","organizations_url":"https://api.github.com/users/albertzaharovits/orgs","repos_url":"https://api.github.com/users/albertzaharovits/repos","events_url":"https://api.github.com/users/albertzaharovits/events{/privacy}","received_events_url":"https://api.github.com/users/albertzaharovits/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2018-06-02T13:05:19Z","updated_at":"2018-06-27T04:03:16Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"To facilitate machine readability, log entries should be structured in a dictionary-like format. Structured logging is also known as semantic or typed logging. Examples of such structured formats are: key value pairs, JSON and XML. In general, flat text entries might be more concise and trade ambiguity for human readability. Machine readable entries are more verbose and explicit, shunning ambiguity.\r\n\r\nThe status quo for the audit logging is a mixture of the templated and the structured entry type.  In general, the templated entry type is a good tradeoff between user readability and the dictionary-like format. It is not ambiguous and is also more concise compared to the structured type. Here is an example entry from the current implementation:\r\n```\r\n[2018-05-30T07:11:39,605] [transport] [access_granted]  origin_type=[local_node], origin_address=[127.0.0.1], principal=[_xpack_security], roles=[superuser], action=[indices:data/read/search[free_context/scroll]]\r\n```\r\nUnfortunately, in the current implementation, entries are ambiguous because the [set of allowed characters](https://github.com/elastic/elasticsearch/blob/5bfe2ba4690c2e55b207c583cff0d8bc98a07988/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/support/Validation.java#L23) in user and role names includes '=', ' ', '[' and ']'. Fixing this would allow for unambiguous templated log entries.\r\n\r\nUnambiguous means machine readable, i.e. you can write a regex to parse a specific field, so why bother with the verbose structured format? The answer is that the next step after parsing is indexing, and indexing requires that event fields have a name and a type. The parser of a templated entry type infers the field name and type from the token's position. But there may be several entry types with different positional parsing required, so one has to create several patterns. But what if patterns overlap (several can match the same line), or more commonly, what if parsers assign the same field name to fields with different data types. This is very possible because the code which generates entries and the code which parses and assigns type names are in different projects. For that matter, it is also relatively easy for the generating and the parsing pieces to go out of sync with each other, requiring integration tests...\r\n\r\n**Lecture over, here is the proposal:**\r\nThe format of the entries should be JSON objects, UTF-8 encoded. This format is dictionary-like, unambiguous, where each field has a name and special characters in values are escaped. The new format will coexist alongside the present one, in an separate file trail.\r\nField names should align to the [elastic common schema (ECS)](https://github.com/elastic/ecs). ECS encourages [dot notation](https://github.com/elastic/ecs#why-is-ecs-using-a-dot-nation-instead-of-an-underline-notation) for field names, i.e. related fields are similarly prefixed. This proposal will follow this suggestion, but note that entries will **not** be nested, values are not dictionaries. \r\nThe stretched goal is to have the format configurable, so even SYSLOG message format from [RFC5424](https://tools.ietf.org/html/rfc5424) might be supported. One possible avenue for this are [messages in log4j 2](https://logging.apache.org/log4j/2.x/manual/messages.html).\r\n\r\nThis proposal is a narrowing of https://github.com/elastic/elasticsearch/issues/8786 to the scope of the audit log.\r\nRelates to https://github.com/elastic/elasticsearch/issues/29881 .\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/400536016","html_url":"https://github.com/elastic/elasticsearch/issues/31046#issuecomment-400536016","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/31046","id":400536016,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMDUzNjAxNg==","user":{"login":"jasontedor","id":4744941,"node_id":"MDQ6VXNlcjQ3NDQ5NDE=","avatar_url":"https://avatars3.githubusercontent.com/u/4744941?v=4","gravatar_id":"","url":"https://api.github.com/users/jasontedor","html_url":"https://github.com/jasontedor","followers_url":"https://api.github.com/users/jasontedor/followers","following_url":"https://api.github.com/users/jasontedor/following{/other_user}","gists_url":"https://api.github.com/users/jasontedor/gists{/gist_id}","starred_url":"https://api.github.com/users/jasontedor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasontedor/subscriptions","organizations_url":"https://api.github.com/users/jasontedor/orgs","repos_url":"https://api.github.com/users/jasontedor/repos","events_url":"https://api.github.com/users/jasontedor/events{/privacy}","received_events_url":"https://api.github.com/users/jasontedor/received_events","type":"User","site_admin":false},"created_at":"2018-06-27T04:03:16Z","updated_at":"2018-06-27T04:03:16Z","author_association":"MEMBER","body":"I am concerned by the fact that this discussion is being driven on a targeted issue (structured logging for audit logging) rather than more broadly within Elasticsearch. Is there a reason that we are approaching this from this angle?"}}
{"actor":{"display_login":"jasontedor"}}