{"action":"created","issue":{"url":"https://api.github.com/repos/fluent/fluent-bit/issues/534","repository_url":"https://api.github.com/repos/fluent/fluent-bit","labels_url":"https://api.github.com/repos/fluent/fluent-bit/issues/534/labels{/name}","comments_url":"https://api.github.com/repos/fluent/fluent-bit/issues/534/comments","events_url":"https://api.github.com/repos/fluent/fluent-bit/issues/534/events","html_url":"https://github.com/fluent/fluent-bit/issues/534","id":304383373,"number":534,"title":"docker json logs using the \"Decode_Field json\" feature cannot be indexed through elasticsearch","user":{"login":"sestrel","id":36937118,"avatar_url":"https://avatars1.githubusercontent.com/u/36937118?v=4","gravatar_id":"","url":"https://api.github.com/users/sestrel","html_url":"https://github.com/sestrel","followers_url":"https://api.github.com/users/sestrel/followers","following_url":"https://api.github.com/users/sestrel/following{/other_user}","gists_url":"https://api.github.com/users/sestrel/gists{/gist_id}","starred_url":"https://api.github.com/users/sestrel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sestrel/subscriptions","organizations_url":"https://api.github.com/users/sestrel/orgs","repos_url":"https://api.github.com/users/sestrel/repos","events_url":"https://api.github.com/users/sestrel/events{/privacy}","received_events_url":"https://api.github.com/users/sestrel/received_events","type":"User","site_admin":false},"labels":[{"id":171021296,"url":"https://api.github.com/repos/fluent/fluent-bit/labels/bug","name":"bug","color":"fc2929","default":true},{"id":644189272,"url":"https://api.github.com/repos/fluent/fluent-bit/labels/fixed","name":"fixed","color":"1d76db","default":false}],"state":"closed","locked":false,"assignee":{"login":"edsiper","id":369718,"avatar_url":"https://avatars3.githubusercontent.com/u/369718?v=4","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false},"assignees":[{"login":"edsiper","id":369718,"avatar_url":"https://avatars3.githubusercontent.com/u/369718?v=4","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/fluent/fluent-bit/milestones/5","html_url":"https://github.com/fluent/fluent-bit/milestone/5","labels_url":"https://api.github.com/repos/fluent/fluent-bit/milestones/5/labels","id":3148964,"number":5,"title":"0.13 Release","description":"","creator":{"login":"edsiper","id":369718,"avatar_url":"https://avatars3.githubusercontent.com/u/369718?v=4","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":5,"state":"open","created_at":"2018-02-27T22:38:30Z","updated_at":"2018-03-23T21:07:32Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2018-03-12T14:01:44Z","updated_at":"2018-03-23T21:07:32Z","closed_at":"2018-03-23T21:07:32Z","author_association":"NONE","body":"I've read the documentation rearding the nested JSON support you are currently implementing for 0.13 in form of parser decoders: http://fluentbit.io/documentation/0.13/parser/decoder.html\r\n\r\nYou describe a typical use case in form of a containerized environments with Docker where the application logs data in JSON format which becomes an escaped string.\r\n\r\nWhen using the \"Decode_Field json log\" there is one problem here, everything written to sysout/syserr is being written to disk by the docker json-file log driver which is being used to accomplish the described use case. But, not every line written to sysout/syserr is a well formatted JSON message. This leads to: \r\n\r\n- if the content is not a valid json-content the \"log\" attribute is of type \"string\"  \r\n\r\n-  if the content is a well formed json the \"log\" attribute is of type object \r\n\r\nSo why is this a problem? We do all that work because we want to collect all those logs and make em searchable through elasticsearch or something else. But the type of a json attribute has to stay the same for being indexable with elasticsearch. It has to stay either a string or an object.\r\n\r\nAre there any plans for a remapping support of the decoded json content? The kubernetes filter for example merges all fields into the parent locument. Here we could do something similiar."},"comment":{"url":"https://api.github.com/repos/fluent/fluent-bit/issues/comments/375798672","html_url":"https://github.com/fluent/fluent-bit/issues/534#issuecomment-375798672","issue_url":"https://api.github.com/repos/fluent/fluent-bit/issues/534","id":375798672,"user":{"login":"edsiper","id":369718,"avatar_url":"https://avatars3.githubusercontent.com/u/369718?v=4","gravatar_id":"","url":"https://api.github.com/users/edsiper","html_url":"https://github.com/edsiper","followers_url":"https://api.github.com/users/edsiper/followers","following_url":"https://api.github.com/users/edsiper/following{/other_user}","gists_url":"https://api.github.com/users/edsiper/gists{/gist_id}","starred_url":"https://api.github.com/users/edsiper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edsiper/subscriptions","organizations_url":"https://api.github.com/users/edsiper/orgs","repos_url":"https://api.github.com/users/edsiper/repos","events_url":"https://api.github.com/users/edsiper/events{/privacy}","received_events_url":"https://api.github.com/users/edsiper/received_events","type":"User","site_admin":false},"created_at":"2018-03-23T21:07:32Z","updated_at":"2018-03-23T21:07:32Z","author_association":"CONTRIBUTOR","body":"Closing as fixed. If you still face the issue we can re open this ticket."}}
{"actor":{"display_login":"edsiper"}}