{"action":"created","issue":{"url":"https://api.github.com/repos/influxdata/telegraf/issues/4759","repository_url":"https://api.github.com/repos/influxdata/telegraf","labels_url":"https://api.github.com/repos/influxdata/telegraf/issues/4759/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/telegraf/issues/4759/comments","events_url":"https://api.github.com/repos/influxdata/telegraf/issues/4759/events","html_url":"https://github.com/influxdata/telegraf/issues/4759","id":364404856,"node_id":"MDU6SXNzdWUzNjQ0MDQ4NTY=","number":4759,"title":"Improve documentation of syslog plugin","user":{"login":"candlerb","id":44789,"node_id":"MDQ6VXNlcjQ0Nzg5","avatar_url":"https://avatars2.githubusercontent.com/u/44789?v=4","gravatar_id":"","url":"https://api.github.com/users/candlerb","html_url":"https://github.com/candlerb","followers_url":"https://api.github.com/users/candlerb/followers","following_url":"https://api.github.com/users/candlerb/following{/other_user}","gists_url":"https://api.github.com/users/candlerb/gists{/gist_id}","starred_url":"https://api.github.com/users/candlerb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/candlerb/subscriptions","organizations_url":"https://api.github.com/users/candlerb/orgs","repos_url":"https://api.github.com/users/candlerb/repos","events_url":"https://api.github.com/users/candlerb/events{/privacy}","received_events_url":"https://api.github.com/users/candlerb/received_events","type":"User","site_admin":false},"labels":[{"id":978508800,"node_id":"MDU6TGFiZWw5Nzg1MDg4MDA=","url":"https://api.github.com/repos/influxdata/telegraf/labels/area/syslog","name":"area/syslog","color":"f48d38","default":false},{"id":561634381,"node_id":"MDU6TGFiZWw1NjE2MzQzODE=","url":"https://api.github.com/repos/influxdata/telegraf/labels/documentation","name":"documentation","color":"c6ffd0","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-09-27T10:23:02Z","updated_at":"2018-09-28T01:12:34Z","closed_at":null,"author_association":"NONE","body":"I was very happy to see the inclusion of a syslog plugin.  However the [documentation](https://github.com/influxdata/telegraf/tree/release-1.8/plugins/inputs/syslog) is missing some critical information, in particular:\r\n\r\n* Does the event timestamp (influxdb timestamp) come from the time the message was received, or the timestamp contained *within* the message?\r\n* What happens for messages which contain the same timestamp?\r\n* Does the `hostname` field contain the source IP address of the syslog message, or the hostname contained *within* the message?\r\n\r\nThe timestamp issue is important because (a) many devices generate syslog messages containing only 1-second resolution timestamps, and (b) influxdb can only store one data point in a timeseries for a particular timestamp.\r\n\r\nSo I set out to find out by experimentation what happens.  I used the sample config which receives TCP on port 6514, and the sample rsyslogd configuration which forwards to it.  Then I did:\r\n\r\n```\r\n# logger foo; logger bar\r\n```\r\n\r\ntcpdump on port 6514 showed the following messages generated by rsyslogd:\r\n\r\n```\r\n60 <13>1 2018-09-27T07:41:24.310199+00:00 tick root - - -  foo\r\n60 <13>1 2018-09-27T07:41:24.314932+00:00 tick root - - -  bar\r\n```\r\n\r\nAnd the results in the database:\r\n\r\n```\r\n> use telegraf\r\nUsing database telegraf\r\n> select * from syslog where message =~ /^ (foo|bar)/\r\nname: syslog\r\ntime                           appname facility facility_code host hostname message procid severity severity_code timestamp           version\r\n----                           ------- -------- ------------- ---- -------- ------- ------ -------- ------------- ---------           -------\r\n2018-09-27T07:41:24.311399024Z root    user     1             tick tick      foo           notice   5             1538034084310199000 1\r\n2018-09-27T07:41:24.31600689Z  root    user     1             tick tick      bar           notice   5             1538034084314932000 1\r\n>\r\n```\r\n\r\nFrom this I infer:\r\n\r\n* The event time (influxdb timestamp) is the time the message was received or processed, with nanosecond resolution\r\n* The `timestamp` field is the time contained *within* the message\r\n\r\nTherefore, unless two messages are processed in the same nanosecond (unlikely), there should be no loss of messages.  However any buffered or backlogged messages will be stored and ordered by time of reception.\r\n\r\nThis all seems fine to me, but it would have been very helpful if this had been documented.\r\n\r\nThe tags and fields are more-or-less in agreement with the documentation:\r\n\r\n```\r\n> show tag keys from syslog\r\nname: syslog\r\ntagKey\r\n------\r\nappname\r\nfacility\r\nhost\r\nhostname\r\nseverity\r\n> show field keys from syslog\r\nname: syslog\r\nfieldKey      fieldType\r\n--------      ---------\r\nfacility_code integer\r\nmessage       string\r\nprocid        string\r\nseverity_code integer\r\ntimestamp     integer\r\nversion       integer\r\n```\r\n\r\nHowever, the documentation says there is a tag \"hostname\", but I also see \"host\".  So to find the difference between these, I did another test:\r\n\r\n```\r\n# echo \"60 <13>1 2018-09-27T07:41:24.310199+00:00 xxxx root - - -  qux\" | nc 127.0.0.1 6514\r\n```\r\n\r\nResult:\r\n\r\n```\r\n> select * from syslog where message =~ /^ qux/\r\nname: syslog\r\ntime                           appname facility facility_code host hostname message procid severity severity_code timestamp           version\r\n----                           ------- -------- ------------- ---- -------- ------- ------ -------- ------------- ---------           -------\r\n2018-09-27T10:18:19.592731705Z root    user     1             tick xxxx      qux           notice   5             1538034084310199000 1\r\n```\r\n\r\nSo it appears that the `hostname` is the hostname contained *within* the message.  But what about `host`?  Is it the source address of the syslog message?\r\n\r\nThe answer is no.  I tried the same nc test from a different host (giving the IP of telegraf instead of 127.0.0.1), and got the same result.\r\n\r\n```\r\n2018-09-27T10:19:56.671438282Z root    user     1             tick xxxx      qux           notice   5             1538034084310199000 1\r\n```\r\n\r\nSo it seems the host is just the host where telegraf itself is running.  Again, would be very helpful if this was documented."},"comment":{"url":"https://api.github.com/repos/influxdata/telegraf/issues/comments/425289117","html_url":"https://github.com/influxdata/telegraf/issues/4759#issuecomment-425289117","issue_url":"https://api.github.com/repos/influxdata/telegraf/issues/4759","id":425289117,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNTI4OTExNw==","user":{"login":"danielnelson","id":1048079,"node_id":"MDQ6VXNlcjEwNDgwNzk=","avatar_url":"https://avatars3.githubusercontent.com/u/1048079?v=4","gravatar_id":"","url":"https://api.github.com/users/danielnelson","html_url":"https://github.com/danielnelson","followers_url":"https://api.github.com/users/danielnelson/followers","following_url":"https://api.github.com/users/danielnelson/following{/other_user}","gists_url":"https://api.github.com/users/danielnelson/gists{/gist_id}","starred_url":"https://api.github.com/users/danielnelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielnelson/subscriptions","organizations_url":"https://api.github.com/users/danielnelson/orgs","repos_url":"https://api.github.com/users/danielnelson/repos","events_url":"https://api.github.com/users/danielnelson/events{/privacy}","received_events_url":"https://api.github.com/users/danielnelson/received_events","type":"User","site_admin":false},"created_at":"2018-09-28T01:12:34Z","updated_at":"2018-09-28T01:12:34Z","author_association":"CONTRIBUTOR","body":"Thanks @candlerb, these are some good observations that will really help getting started with this plugin.\r\n\r\n> maybe there is buffering/batching.\r\n\r\nTelegraf writes in batches of `metric_batch_size` which is 1000 by default.\r\n\r\nAs mentioned on #4762, the `host` tag is a tag we set on all metrics by default, and represents the host running Telegraf.  I'll comment further on the other issue on what we should do with hostname.\r\n\r\nWe will improve the structured data and timestamp documentation as suggested."}}
{"actor":{"display_login":"danielnelson"}}