{"action":"closed","issue":{"url":"https://api.github.com/repos/influxdata/telegraf/issues/1293","repository_url":"https://api.github.com/repos/influxdata/telegraf","labels_url":"https://api.github.com/repos/influxdata/telegraf/issues/1293/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/telegraf/issues/1293/comments","events_url":"https://api.github.com/repos/influxdata/telegraf/issues/1293/events","html_url":"https://github.com/influxdata/telegraf/issues/1293","id":157392156,"number":1293,"title":"flush_interval / flush_jitter  not doing what expected","user":{"login":"daviesalex","id":1106249,"avatar_url":"https://avatars.githubusercontent.com/u/1106249?v=3","gravatar_id":"","url":"https://api.github.com/users/daviesalex","html_url":"https://github.com/daviesalex","followers_url":"https://api.github.com/users/daviesalex/followers","following_url":"https://api.github.com/users/daviesalex/following{/other_user}","gists_url":"https://api.github.com/users/daviesalex/gists{/gist_id}","starred_url":"https://api.github.com/users/daviesalex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/daviesalex/subscriptions","organizations_url":"https://api.github.com/users/daviesalex/orgs","repos_url":"https://api.github.com/users/daviesalex/repos","events_url":"https://api.github.com/users/daviesalex/events{/privacy}","received_events_url":"https://api.github.com/users/daviesalex/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-29T15:47:30Z","updated_at":"2016-05-30T14:50:39Z","closed_at":"2016-05-30T14:50:39Z","body":"We have set this:\r\n\r\n```\r\n[agent]\r\n  flush_interval = \"10s\"\r\n  flush_jitter = \"10s\"\r\n```\r\n\r\nWhat we want is that telegraf flushes every ten seconds, +/- 10 seconds. With enough nodes, this will result in a uniform load and avoid any microbursts.\r\n\r\nWhat we actually see is that each agent picks a random time, and sticks to it. This is bad:\r\n* Even at our scale, this isnt resulting in a uniform distribution as picking a random time per flush would\r\n* Its 16.00 seconds. We have microbursts on the second (our timesync is very good). The mean difference between packets in tcpdump in this case is 16.001427000 seconds and variance minimal.\r\n\r\nHere is a tcpdump of the server showing an even set of packets coming in from one client:\r\n![image](https://cloud.githubusercontent.com/assets/1106249/15634488/00f0ffb2-25bd-11e6-830f-9b2b24279064.png)\r\n\r\nIf we plot the time between packets in wireshark, we see its exactly 16,00 seconds for each burst.\r\n\r\nThe client log shows the same:\r\n```\r\n[root@host1~]# grep \"wrote batch\" /var/log/telegraf/telegraf.log | tail\r\n2016/05/29 07:28:22 Output [influxdb] wrote batch of 32 metrics in 319.224µs\r\n2016/05/29 07:28:22 Output [influxdb] wrote batch of 32 metrics in 413.21µs\r\n2016/05/29 07:28:38 Output [influxdb] wrote batch of 32 metrics in 264.434µs\r\n2016/05/29 07:28:38 Output [influxdb] wrote batch of 32 metrics in 261.492µs\r\n2016/05/29 07:28:54 Output [influxdb] wrote batch of 25 metrics in 253.743µs\r\n2016/05/29 07:28:54 Output [influxdb] wrote batch of 25 metrics in 374.704µs\r\n2016/05/29 07:29:10 Output [influxdb] wrote batch of 23 metrics in 322.524µs\r\n2016/05/29 07:29:10 Output [influxdb] wrote batch of 23 metrics in 487.075µs\r\n2016/05/29 07:29:26 Output [influxdb] wrote batch of 32 metrics in 482.082µs\r\n2016/05/29 07:29:26 Output [influxdb] wrote batch of 32 metrics in 729.194µs\r\n```\r\n22 -> 38 -> 54 -> 10 -> 26 is obviously exactly 16.\r\n\r\nAnother client shows 10:\r\n```\r\n[root@host2 ~]# grep \"wrote batch\" /var/log/telegraf/telegraf.log | tail\r\n2016/05/29 07:35:03 Output [influxdb] wrote batch of 16 metrics in 183.542µs\r\n2016/05/29 07:35:03 Output [influxdb] wrote batch of 16 metrics in 184.473µs\r\n2016/05/29 07:35:13 Output [influxdb] wrote batch of 16 metrics in 172.352µs\r\n2016/05/29 07:35:13 Output [influxdb] wrote batch of 16 metrics in 159.924µs\r\n2016/05/29 07:35:23 Output [influxdb] wrote batch of 16 metrics in 169.777µs\r\n2016/05/29 07:35:23 Output [influxdb] wrote batch of 16 metrics in 139.105µs\r\n2016/05/29 07:35:33 Output [influxdb] wrote batch of 16 metrics in 111.461µs\r\n2016/05/29 07:35:33 Output [influxdb] wrote batch of 16 metrics in 257.57µs\r\n2016/05/29 07:35:43 Output [influxdb] wrote batch of 22 metrics in 171.405µs\r\n2016/05/29 07:35:43 Output [influxdb] wrote batch of 22 metrics in 189.031µs\r\n```\r\nAnd another 20:\r\n```\r\n[root@host3 ~]# grep \"wrote batch\" /var/log/telegraf/telegraf.log | tail\r\n2016/05/29 07:34:40 Output [influxdb] wrote batch of 32 metrics in 370.291µs\r\n2016/05/29 07:34:40 Output [influxdb] wrote batch of 32 metrics in 403.677µs\r\n2016/05/29 07:34:59 Output [influxdb] wrote batch of 32 metrics in 226.281µs\r\n2016/05/29 07:34:59 Output [influxdb] wrote batch of 32 metrics in 241.475µs\r\n2016/05/29 07:35:19 Output [influxdb] wrote batch of 42 metrics in 239.915µs\r\n2016/05/29 07:35:19 Output [influxdb] wrote batch of 42 metrics in 221.367µs\r\n2016/05/29 07:35:39 Output [influxdb] wrote batch of 38 metrics in 195.99µs\r\n2016/05/29 07:35:39 Output [influxdb] wrote batch of 38 metrics in 221.801µs\r\n2016/05/29 07:35:58 Output [influxdb] wrote batch of 32 metrics in 195.303µs\r\n2016/05/29 07:35:58 Output [influxdb] wrote batch of 32 metrics in 196.709µs\r\n```\r\n\r\nWhat we want is that for this period to be randomized (by the jitter value) for each burst, and that randomization to be to the nanosecond (not second) so its truly random."}}
{"actor":{"display_login":"kostasb"}}