{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/1249","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/1249/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/1249/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/1249/events","html_url":"https://github.com/hashicorp/consul/issues/1249","id":107373536,"number":1249,"title":"logs would be more useful if they were structured","user":{"login":"macb","id":1611098,"avatar_url":"https://avatars.githubusercontent.com/u/1611098?v=3","gravatar_id":"","url":"https://api.github.com/users/macb","html_url":"https://github.com/macb","followers_url":"https://api.github.com/users/macb/followers","following_url":"https://api.github.com/users/macb/following{/other_user}","gists_url":"https://api.github.com/users/macb/gists{/gist_id}","starred_url":"https://api.github.com/users/macb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/macb/subscriptions","organizations_url":"https://api.github.com/users/macb/orgs","repos_url":"https://api.github.com/users/macb/repos","events_url":"https://api.github.com/users/macb/events{/privacy}","received_events_url":"https://api.github.com/users/macb/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hashicorp/consul/labels/thinking","name":"thinking","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"slackpad","id":673509,"avatar_url":"https://avatars.githubusercontent.com/u/673509?v=3","gravatar_id":"","url":"https://api.github.com/users/slackpad","html_url":"https://github.com/slackpad","followers_url":"https://api.github.com/users/slackpad/followers","following_url":"https://api.github.com/users/slackpad/following{/other_user}","gists_url":"https://api.github.com/users/slackpad/gists{/gist_id}","starred_url":"https://api.github.com/users/slackpad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slackpad/subscriptions","organizations_url":"https://api.github.com/users/slackpad/orgs","repos_url":"https://api.github.com/users/slackpad/repos","events_url":"https://api.github.com/users/slackpad/events{/privacy}","received_events_url":"https://api.github.com/users/slackpad/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2015-09-20T05:13:26Z","updated_at":"2016-02-02T07:52:15Z","closed_at":null,"body":"The logging in consul is quite useful already, especially when exploring bits and pieces with debug/trace. However, the format is really only useful to humans and/or programs intimately familiar with consul's logs. It would be incredibly useful if the format was configurable and one such format offered more computer parseable structure.\r\n\r\nAs an example:\r\n\r\nhttps://github.com/hashicorp/consul/blob/master/command/agent/watch_handler.go#L43\r\n```\r\nlogger.Printf(\"[ERR] agent: Failed to setup watch: %v\", err)\r\n```\r\n\r\nThis line contains a lot of information:\r\n  - The level: We're logging an error.\r\n  - Where we are: The error is coming from the agent subsystem\r\n  - The impact: We failed to setup a watch\r\n  - What happened: The error itself.\r\n\r\nIn order to parse this log line we'd need to strip off the level, pull out what system we're in, take out the identifiable string (\"failed to setup watch\") and also store the error that arose. This is relatively easy to do as humans but computers are a little less flexible.\r\n\r\nIf instead we used a structured logging format like JSON we could end up with a log like:\r\n```json\r\n{\r\n  \"level\": \"error\",\r\n  \"system\": \"agent\",\r\n  \"error\": \"the error string from the error that occured\",\r\n  \"msg\": \"failed to setup watch\" \r\n}\r\n```\r\n\r\nAnd to produce that output we could change our interface accordingly:\r\n```\r\n// setup early and re-used as needed\r\nlogger := log.WithField(\"system\", \"agent\")\r\n// Do some stuff\r\nif err != nil {\r\n    logger.WithError(err).Error(\"failed to setup watch\")\r\n}\r\n```\r\nBreaking this interface down: \r\n - `.WithField` could add an arbitrary k/v pair to the log. It returns a new structure so that we could chain them together without altering the parent.\r\n - `.WithError` might be a convenience method for consistencies sake and creates an `\"error\"` key with the value being the error's value.\r\n - `.Error` is the print method itself which serves two purposes. It creates the \"message\" we want to send (the consistent string we can depend on always existing in this log line) as well as setup the level that we want to print at. The underlying writer interface could then be responsible for deciding whether or not we should _actually_ print the log based on the log level we're configured to.\r\n\r\nOn top of homegrown solutions structured logs can play quite nicely with systems like [rsyslog](http://www.rsyslog.com/tag/cee/). A very popular generic library for this flow is https://github.com/Sirupsen/logrus. There's no doubt this would be a pretty big change which is why I wanted to open the issue and get input before going rogue."},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/178431405","html_url":"https://github.com/hashicorp/consul/issues/1249#issuecomment-178431405","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/1249","id":178431405,"user":{"login":"bastichelaar","id":1071127,"avatar_url":"https://avatars.githubusercontent.com/u/1071127?v=3","gravatar_id":"","url":"https://api.github.com/users/bastichelaar","html_url":"https://github.com/bastichelaar","followers_url":"https://api.github.com/users/bastichelaar/followers","following_url":"https://api.github.com/users/bastichelaar/following{/other_user}","gists_url":"https://api.github.com/users/bastichelaar/gists{/gist_id}","starred_url":"https://api.github.com/users/bastichelaar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bastichelaar/subscriptions","organizations_url":"https://api.github.com/users/bastichelaar/orgs","repos_url":"https://api.github.com/users/bastichelaar/repos","events_url":"https://api.github.com/users/bastichelaar/events{/privacy}","received_events_url":"https://api.github.com/users/bastichelaar/received_events","type":"User","site_admin":false},"created_at":"2016-02-02T07:52:15Z","updated_at":"2016-02-02T07:52:15Z","body":"Any progress on this? Structured logging would indeed be very useful!"}}