{"action":"created","issue":{"url":"https://api.github.com/repos/influxdata/kapacitor/issues/939","repository_url":"https://api.github.com/repos/influxdata/kapacitor","labels_url":"https://api.github.com/repos/influxdata/kapacitor/issues/939/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/kapacitor/issues/939/comments","events_url":"https://api.github.com/repos/influxdata/kapacitor/issues/939/events","html_url":"https://github.com/influxdata/kapacitor/issues/939","id":178901395,"number":939,"title":"Deadman - False Triggers","user":{"login":"octalthorpe","id":6535531,"avatar_url":"https://avatars.githubusercontent.com/u/6535531?v=3","gravatar_id":"","url":"https://api.github.com/users/octalthorpe","html_url":"https://github.com/octalthorpe","followers_url":"https://api.github.com/users/octalthorpe/followers","following_url":"https://api.github.com/users/octalthorpe/following{/other_user}","gists_url":"https://api.github.com/users/octalthorpe/gists{/gist_id}","starred_url":"https://api.github.com/users/octalthorpe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/octalthorpe/subscriptions","organizations_url":"https://api.github.com/users/octalthorpe/orgs","repos_url":"https://api.github.com/users/octalthorpe/repos","events_url":"https://api.github.com/users/octalthorpe/events{/privacy}","received_events_url":"https://api.github.com/users/octalthorpe/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdata/kapacitor/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/influxdata/kapacitor/labels/deadman","name":"deadman","color":"b60205"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/influxdata/kapacitor/milestones/9","html_url":"https://github.com/influxdata/kapacitor/milestones/v1.1.0","labels_url":"https://api.github.com/repos/influxdata/kapacitor/milestones/9/labels","id":1814111,"number":9,"title":"v1.1.0","description":"","creator":{"login":"nathanielc","id":3771906,"avatar_url":"https://avatars.githubusercontent.com/u/3771906?v=3","gravatar_id":"","url":"https://api.github.com/users/nathanielc","html_url":"https://github.com/nathanielc","followers_url":"https://api.github.com/users/nathanielc/followers","following_url":"https://api.github.com/users/nathanielc/following{/other_user}","gists_url":"https://api.github.com/users/nathanielc/gists{/gist_id}","starred_url":"https://api.github.com/users/nathanielc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathanielc/subscriptions","organizations_url":"https://api.github.com/users/nathanielc/orgs","repos_url":"https://api.github.com/users/nathanielc/repos","events_url":"https://api.github.com/users/nathanielc/events{/privacy}","received_events_url":"https://api.github.com/users/nathanielc/received_events","type":"User","site_admin":false},"open_issues":8,"closed_issues":3,"state":"open","created_at":"2016-06-07T18:42:51Z","updated_at":"2016-09-26T18:11:25Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2016-09-23T15:07:21Z","updated_at":"2016-09-29T16:12:35Z","closed_at":null,"body":"There seem to already be a few issues open on the deadman firing falsely.  In my case I even see times where the deadman fires indicating it received a single point, which should have meant it would have passed that check.\r\n\r\n```\r\n\"moarcanary-auth\": No Data Check - CRITICAL\r\n           1.00 points received in 5m \r\n```\r\nand in the logfile, it appears there is at least 1 point\r\n```\r\n{\r\n  \"id\": \"nil:where2 for task 'moarcanary-auth'\",\r\n  \"message\": \"\\\"moarcanary-auth\\\": No Data Check - CRITICAL\\n            1.00 points received in 5m\",\r\n  \"details\": \"{&#34;Name&#34;:&#34;stats&#34;,&#34;TaskName&#34;:&#34;moarcanary-auth&#34;,&#34;Group&#34;:&#34;nil&#34;,&#34;Tags&#34;:null,&#34;ID&#34;:&#34;nil:where2 for task &#39;moarcanary-auth&#39;&#34;,&#34;Fields&#34;:{&#34;emitted&#34;:1},&#34;Level&#34;:&#34;CRITICAL&#34;,&#34;Time&#34;:&#34;2016-09-23T14:50:00Z&#34;,&#34;Message&#34;:&#34;\\\\&#34;moarcanary-auth\\\\&#34;: No Data Check - CRITICAL\\\\n            1.00 points recieved in 5m&#34;}\\n\",\r\n  \"time\": \"2016-09-23T14:50:00Z\",\r\n  \"duration\": 0,\r\n  \"level\": \"CRITICAL\",\r\n  \"data\": {\r\n    \"series\": [\r\n      {\r\n        \"name\": \"stats\",\r\n        \"columns\": [\r\n          \"time\",\r\n          \"emitted\"\r\n        ],\r\n        \"values\": [\r\n          [\r\n            \"2016-09-23T14:50:00Z\",\r\n            1\r\n          ]\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}\r\n```\r\n\r\nMy tick script is as follows\r\n```\r\nvar data = stream\r\n    |from()\r\n        .database(database)\r\n        .retentionPolicy(retention_policy)\r\n        .measurement(metric_name)\r\n    |where(lambda: \"deployment\" == deployment)\r\n    //|log()\r\n\r\ndata\r\n    |deadman(1.0, 5m)\r\n        .message('\"{{ .TaskName }}\": No Data Check - {{.Level}}\r\n            {{ index .Fields \"emitted\" | printf \"%0.2f\" }} points received in 5m')\r\n        .log('/tmp/moarcarnay.log')\r\n        .slack()\r\n        .channel(slack_channel)\r\ndata\r\n    |window()\r\n        .period(5m)\r\n        .every(30s)\r\n    |mean('value')\r\n        .as('mean_value')\r\n    //Slack\r\n    |alert()\r\n        .stateChangesOnly()\r\n        .message('{{.TaskName}} {{ .Level }}\r\n            Got a value of {{ index .Fields \"mean_value\" }} -- Expecting == 1.0')\r\n        .warn(lambda: \"mean_value\" < 1.0)\r\n        .crit(lambda: \"mean_value\" < 1.0)\r\n        .slack()\r\n          .channel(slack_channel)\r\n    //OpsGenie\r\n    |alert()\r\n        .stateChangesOnly()\r\n        .message('{{ .TaskName }} {{ .Level }}\r\n            Got a value of {{ index .Fields \"mean_value\" }} -- Expecting == 1.0')\r\n        .crit(lambda: \"mean_value\" < 1.0 AND callout == TRUE)\r\n        .slack()\r\n          .channel(slack_channel)\r\n        .opsGenie()\r\n             .teams(opsgenie_team)\r\n             .recipients([ opsgenie_recipients ])\r\n```\r\n\r\nIn other cases I see that I received 10 points in 5m, which is double the possible number I should receive. \r\n```\r\n{\r\n  \"id\": \"nil:where2 for task 'moarcanary-auth'\",\r\n  \"message\": \"\\\"moarcanary-auth\\\": No Data Check - OK\\n            10.00 points recieved in 5m\",\r\n  \"details\": \"{&#34;Name&#34;:&#34;stats&#34;,&#34;TaskName&#34;:&#34;moarcanary-auth&#34;,&#34;Group&#34;:&#34;nil&#34;,&#34;Tags&#34;:{&#34;deployment&#34;:&#34;staging&#34;,&#34;host&#34;:&#34;ecs-a-ancillary-0-002.internal.staging.iot.blackberry.com&#34;,&#34;metric_type&#34;:&#34;gauge&#34;,&#34;region&#34;:&#34;us-east-1&#34;},&#34;ID&#34;:&#34;nil:where2 for task &#39;moarcanary-auth&#39;&#34;,&#34;Fields&#34;:{&#34;emitted&#34;:10},&#34;Level&#34;:&#34;OK&#34;,&#34;Time&#34;:&#34;2016-09-23T14:55:00Z&#34;,&#34;Message&#34;:&#34;\\\\&#34;moarcanary-auth\\\\&#34;: No Data Check - OK\\\\n            10.00 points recieved in 5m&#34;}\\n\",\r\n  \"time\": \"2016-09-23T14:55:00Z\",\r\n  \"duration\": 300000000000,\r\n  \"level\": \"OK\",\r\n  \"data\": {\r\n    \"series\": [\r\n      {\r\n        \"name\": \"stats\",\r\n        \"tags\": {\r\n          \"deployment\": \"staging\",\r\n          \"host\": \"ecs-a-ancillary-0-002.internal.staging.iot.blackberry.com\",\r\n          \"metric_type\": \"gauge\",\r\n          \"region\": \"us-east-1\"\r\n        },\r\n        \"columns\": [\r\n          \"time\",\r\n          \"emitted\"\r\n        ],\r\n        \"values\": [\r\n          [\r\n            \"2016-09-23T14:55:00Z\",\r\n            10\r\n          ]\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}\r\n```\r\n\r\nI am using the HTTP subscription method and influxdb 1.0.0/ kapacitor 1.0.0\r\n\r\nThe other issues open don't seem to have a resolution so figured I would share my situation as it is of a slightly different pattern.\r\n\r\nI will continue to see if I can debug this further to aid in the resolution."},"comment":{"url":"https://api.github.com/repos/influxdata/kapacitor/issues/comments/250513990","html_url":"https://github.com/influxdata/kapacitor/issues/939#issuecomment-250513990","issue_url":"https://api.github.com/repos/influxdata/kapacitor/issues/939","id":250513990,"user":{"login":"octalthorpe","id":6535531,"avatar_url":"https://avatars.githubusercontent.com/u/6535531?v=3","gravatar_id":"","url":"https://api.github.com/users/octalthorpe","html_url":"https://github.com/octalthorpe","followers_url":"https://api.github.com/users/octalthorpe/followers","following_url":"https://api.github.com/users/octalthorpe/following{/other_user}","gists_url":"https://api.github.com/users/octalthorpe/gists{/gist_id}","starred_url":"https://api.github.com/users/octalthorpe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/octalthorpe/subscriptions","organizations_url":"https://api.github.com/users/octalthorpe/orgs","repos_url":"https://api.github.com/users/octalthorpe/repos","events_url":"https://api.github.com/users/octalthorpe/events{/privacy}","received_events_url":"https://api.github.com/users/octalthorpe/received_events","type":"User","site_admin":false},"created_at":"2016-09-29T16:12:35Z","updated_at":"2016-09-29T16:12:35Z","body":"I should be able to get to reproducing tomorrow, and update with my findings."}}
{"actor":{"display_login":"octalthorpe"}}