{"action":"created","issue":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/5951","repository_url":"https://api.github.com/repos/influxdata/influxdb","labels_url":"https://api.github.com/repos/influxdata/influxdb/issues/5951/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/influxdb/issues/5951/comments","events_url":"https://api.github.com/repos/influxdata/influxdb/issues/5951/events","html_url":"https://github.com/influxdata/influxdb/issues/5951","id":139628769,"number":5951,"title":"[0.10.2] Continuous queries slowly 'fading away'","user":{"login":"KiNgMaR","id":438358,"avatar_url":"https://avatars.githubusercontent.com/u/438358?v=3","gravatar_id":"","url":"https://api.github.com/users/KiNgMaR","html_url":"https://github.com/KiNgMaR","followers_url":"https://api.github.com/users/KiNgMaR/followers","following_url":"https://api.github.com/users/KiNgMaR/following{/other_user}","gists_url":"https://api.github.com/users/KiNgMaR/gists{/gist_id}","starred_url":"https://api.github.com/users/KiNgMaR/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KiNgMaR/subscriptions","organizations_url":"https://api.github.com/users/KiNgMaR/orgs","repos_url":"https://api.github.com/users/KiNgMaR/repos","events_url":"https://api.github.com/users/KiNgMaR/events{/privacy}","received_events_url":"https://api.github.com/users/KiNgMaR/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdata/influxdb/labels/category/continuous%20queries","name":"category/continuous queries","color":"207de5"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-03-09T16:32:51Z","updated_at":"2016-06-30T08:13:32Z","closed_at":null,"body":"Hi everyone,\r\n\r\nwe suffer from the following issue in a single-node setup. Some meta information:\r\n\r\n* 2016/03/09 07:37:12 InfluxDB starting, version 0.10.2, branch 0.10.0, commit 845ce867189724762a6637f17da742f2ca0a9625, built 2016-03-03T22:05:35.507486\r\n* 2016/03/09 07:37:12 Go version go1.4.3, GOMAXPROCS set to 8\r\n* CentOS 6.7\r\n* 8 CPU Cores\r\n* 16 GB RAM\r\n\r\nI wiped all meta data and data this morning and performed a clean install of 0.10.2. After that, I created 7 continuous queries as follows:\r\n\r\n* cq_xxx_rows_per_minute_total\t\"CREATE CONTINUOUS QUERY cq_xxx_rows_per_minute_total ON xxx_prod RESAMPLE EVERY 30s FOR 1m BEGIN SELECT sum(num_xxxs) INTO xxx_prod.\"default\".cq_xxx_rows_per_minute_total FROM xxx_prod.\"default\".xxx_requests GROUP BY time(1m) END\"\r\n* cq_yyy_requests_per_minute_method\t\"CREATE CONTINUOUS QUERY cq_yyy_requests_per_minute_method ON xxx_prod RESAMPLE EVERY 30s FOR 1m BEGIN SELECT count(\"duration\") INTO xxx_prod.\"default\".cq_yyy_requests_per_minute_method FROM xxx_prod.\"default\".yyy_requests GROUP BY time(1m), method END\"\r\n* cq_xxx_requests_per_minute_server\t\"CREATE CONTINUOUS QUERY cq_xxx_requests_per_minute_server ON xxx_prod RESAMPLE EVERY 30s FOR 1m BEGIN SELECT count(\"duration\") INTO xxx_prod.\"default\".cq_xxx_requests_per_minute_server FROM xxx_prod.\"default\".xxx_requests GROUP BY time(1m), \"server\" END\"\r\n* cq_xxx_requests_mean_duration_15s\t\"CREATE CONTINUOUS QUERY cq_xxx_requests_mean_duration_15s ON xxx_prod RESAMPLE EVERY 15s FOR 30s BEGIN SELECT mean(\"duration\") INTO xxx_prod.\"default\".cq_xxx_requests_mean_duration_15s FROM xxx_prod.\"default\".xxx_requests WHERE error = '0' GROUP BY time(15s) END\"\r\n* cq_xxx_requests_min_duration_15s\t\"CREATE CONTINUOUS QUERY cq_xxx_requests_min_duration_15s ON xxx_prod RESAMPLE EVERY 15s FOR 30s BEGIN SELECT min(\"duration\") INTO xxx_prod.\"default\".cq_xxx_requests_min_duration_15s FROM xxx_prod.\"default\".xxx_requests WHERE error = '0' GROUP BY time(15s) END\"\r\n* cq_xxx_requests_max_duration_15s\t\"CREATE CONTINUOUS QUERY cq_xxx_requests_max_duration_15s ON xxx_prod RESAMPLE EVERY 15s FOR 30s BEGIN SELECT max(\"duration\") INTO xxx_prod.\"default\".cq_xxx_requests_max_duration_15s FROM xxx_prod.\"default\".xxx_requests WHERE error = '0' GROUP BY time(15s) END\"\r\n* cq_xxx_requests_95pct_duration_15s\t\"CREATE CONTINUOUS QUERY cq_xxx_requests_95pct_duration_15s ON xxx_prod RESAMPLE EVERY 15s FOR 30s BEGIN SELECT percentile(\"duration\", 95.000) INTO xxx_prod.\"default\".cq_xxx_requests_95pct_duration_15s FROM xxx_prod.\"default\".xxx_requests WHERE error = '0' GROUP BY time(15s) END\"\r\n\r\nFor the first few minutes, all queries have been executing as desired:\r\n\r\n* [continuous_querier] 2016/03/09 07:42:30 executing continuous query cq_xxx_requests_95pct_duration_15s (2016-03-09 07:42:15 +0100 CET to 2016-03-09 07:42:30 +0100 CET)\r\n* [continuous_querier] 2016/03/09 07:42:30 executing continuous query cq_xxx_requests_95pct_duration_15s (2016-03-09 07:42:00 +0100 CET to 2016-03-09 07:42:15 +0100 CET)\r\n* [continuous_querier] 2016/03/09 07:42:45 executing continuous query cq_xxx_requests_mean_duration_15s (2016-03-09 07:42:30 +0100 CET to 2016-03-09 07:42:45 +0100 CET)\r\n* [continuous_querier] 2016/03/09 07:42:45 executing continuous query cq_xxx_requests_mean_duration_15s (2016-03-09 07:42:15 +0100 CET to 2016-03-09 07:42:30 +0100 CET)\r\n\r\nAs we can see, the timestamps in the log and the time spans for the query execution line up nicely.\r\n\r\nFast-forward 5 hours.\r\n\r\n* [continuous_querier] 2016/03/09 12:42:22 executing continuous query cq_xxx_requests_95pct_duration_15s (2016-03-09 12:41:15 +0100 CET to 2016-03-09 12:41:30 +0100 CET)\r\n* [continuous_querier] 2016/03/09 12:42:24 executing continuous query cq_xxx_requests_95pct_duration_15s (2016-03-09 12:41:00 +0100 CET to 2016-03-09 12:41:15 +0100 CET)\r\n* [continuous_querier] 2016/03/09 12:42:26 executing continuous query cq_xxx_requests_95pct_duration_15s (2016-03-09 12:40:45 +0100 CET to 2016-03-09 12:41:00 +0100 CET)\r\n* [continuous_querier] 2016/03/09 12:42:28 executing continuous query cq_xxx_requests_95pct_duration_15s (2016-03-09 12:40:30 +0100 CET to 2016-03-09 12:40:45 +0100 CET)\r\n* [continuous_querier] 2016/03/09 12:42:32 executing continuous query cq_xxx_rows_per_minute_total (2016-03-09 12:42:00 +0100 CET to 2016-03-09 12:43:00 +0100 CET)\r\n* [continuous_querier] 2016/03/09 12:42:34 executing continuous query cq_xxx_rows_per_minute_total (2016-03-09 12:41:00 +0100 CET to 2016-03-09 12:42:00 +0100 CET)\r\n\r\nWe can observe that the timestamps in the log and the timestamps for the execution have drifted apart.\r\n\r\nFast-forward another 5 hours.\r\n\r\n* [continuous_querier] 2016/03/09 17:13:12 executing continuous query cq_xxx_requests_min_duration_15s (2016-03-09 16:25:00 +0100 CET to 2016-03-09 16:25:15 +0100 CET)\r\n* [continuous_querier] 2016/03/09 17:13:18 executing continuous query cq_xxx_requests_min_duration_15s (2016-03-09 16:24:45 +0100 CET to 2016-03-09 16:25:00 +0100 CET)\r\n* [continuous_querier] 2016/03/09 17:13:24 executing continuous query cq_xxx_requests_max_duration_15s (2016-03-09 16:49:00 +0100 CET to 2016-03-09 16:49:15 +0100 CET)\r\n* [continuous_querier] 2016/03/09 17:13:29 executing continuous query cq_xxx_requests_max_duration_15s (2016-03-09 16:48:45 +0100 CET to 2016-03-09 16:49:00 +0100 CET)\r\n\r\nNow the timestamps are completely out of sync. InfluxDB appears to be recalculating data from 50 minutes ago.\r\n\r\nNow let's check the frequency of log entries like:\r\n\r\n     [meta] 2016/03/09 16:49:15 127.0.0.1 - - [09/Mar/2016:16:49:15 +0100] GET /lease?name=continuous_querier&nodeid=1 HTTP/1.1 200 108 - Go 1.1 package http 79bb3505-e60e-11e5-bd7d-000000000000 327.56µs\r\n\r\nA command like `grep lease /var/log/influxdb/influxd.log | grep -oP ' \\d\\d:\\d\\d:\\d\\d 127' | cut -d ':' -f 1-2 | sort | uniq -c` shows the following output (full here: https://gist.github.com/KiNgMaR/c44a86b9b534e49ee00b):\r\n\r\n     25  07:41\r\n     66  07:42\r\n     84  07:43\r\n     59  07:44\r\n     59  07:45\r\n     59  07:46\r\n     59  07:47\r\n     59  07:48\r\n     59  07:49\r\n     59  07:50\r\n     59  07:51\r\n     59  07:52\r\n     59  07:53\r\n     58  07:54\r\n     59  07:55\r\n     58  07:56\r\n    ...\r\n      1  14:55\r\n      1  14:58\r\n      1  15:02\r\n      1  15:05\r\n      1  15:09\r\n      1  15:13\r\n      1  15:17\r\n      1  15:22\r\n      1  15:28\r\n      1  15:34\r\n      1  15:40\r\n      1  15:48\r\n      1  15:57\r\n      1  16:09\r\n      1  16:25\r\n      1  16:49\r\n\r\nI presume this is related to the issue.\r\n\r\nPlease let me know what I can do to assist in tracking this very inconvenient problem down.\r\n\r\nThanks!"},"comment":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/comments/229592142","html_url":"https://github.com/influxdata/influxdb/issues/5951#issuecomment-229592142","issue_url":"https://api.github.com/repos/influxdata/influxdb/issues/5951","id":229592142,"user":{"login":"KiNgMaR","id":438358,"avatar_url":"https://avatars.githubusercontent.com/u/438358?v=3","gravatar_id":"","url":"https://api.github.com/users/KiNgMaR","html_url":"https://github.com/KiNgMaR","followers_url":"https://api.github.com/users/KiNgMaR/followers","following_url":"https://api.github.com/users/KiNgMaR/following{/other_user}","gists_url":"https://api.github.com/users/KiNgMaR/gists{/gist_id}","starred_url":"https://api.github.com/users/KiNgMaR/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KiNgMaR/subscriptions","organizations_url":"https://api.github.com/users/KiNgMaR/orgs","repos_url":"https://api.github.com/users/KiNgMaR/repos","events_url":"https://api.github.com/users/KiNgMaR/events{/privacy}","received_events_url":"https://api.github.com/users/KiNgMaR/received_events","type":"User","site_admin":false},"created_at":"2016-06-30T08:13:32Z","updated_at":"2016-06-30T08:13:32Z","body":"Hey Sean, I can tell you once #6909 is implemented. :) When executed manually, each of the queries takes less than the interval time though."}}
{"actor":{"display_login":"KiNgMaR"}}