{"action":"created","issue":{"url":"https://api.github.com/repos/linkedin/Burrow/issues/123","repository_url":"https://api.github.com/repos/linkedin/Burrow","labels_url":"https://api.github.com/repos/linkedin/Burrow/issues/123/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/Burrow/issues/123/comments","events_url":"https://api.github.com/repos/linkedin/Burrow/issues/123/events","html_url":"https://github.com/linkedin/Burrow/issues/123","id":173471717,"number":123,"title":"Unexplained Error Status","user":{"login":"326TimesBetter","id":16246896,"avatar_url":"https://avatars.githubusercontent.com/u/16246896?v=3","gravatar_id":"","url":"https://api.github.com/users/326TimesBetter","html_url":"https://github.com/326TimesBetter","followers_url":"https://api.github.com/users/326TimesBetter/followers","following_url":"https://api.github.com/users/326TimesBetter/following{/other_user}","gists_url":"https://api.github.com/users/326TimesBetter/gists{/gist_id}","starred_url":"https://api.github.com/users/326TimesBetter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/326TimesBetter/subscriptions","organizations_url":"https://api.github.com/users/326TimesBetter/orgs","repos_url":"https://api.github.com/users/326TimesBetter/repos","events_url":"https://api.github.com/users/326TimesBetter/events{/privacy}","received_events_url":"https://api.github.com/users/326TimesBetter/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-08-26T14:51:06Z","updated_at":"2016-08-26T21:53:15Z","closed_at":null,"body":"We have been running burrow and publishing its logs into kibana, where we are using a tool called elastalert to alert off of them. However, we are getting what appear to me to be a number of false positives that I do not understand. Here are two examples:\r\n\r\n![image](https://cloud.githubusercontent.com/assets/16246896/18009589/ed61520e-6b7a-11e6-8abf-48e8f51f54fe.png)\r\n\r\nThe first row is an example of a log that seems to have had a few logs come in over the time window, and has consumed them (because end log lag is zero).\r\n\r\nThe second row is an example of a log which has had no logs come in over the time window, but remains caught up at end_log_lag zero.\r\n\r\nYet both of these  appear to have a status of \"ERR\".  By the first rule in your \"Consumer Lag Evaluation Section\":`If any lag within the window is zero, the status is considered to be OK.\r\n`, shouldn't these rows be marked status: ok ??\r\n\r\nHere are the full log lines produced by burrow:\r\n\r\n    2016/08/26 14:34:48 ;source_timestamp=2016-08-26T14:34:48Z;cluster=local;consumer_group=cd-changeevent-consumer;error=false;status=ERR;complete=true;lag_type=Partition;topic=directory-users-processed;partition=0;partition_status=STOP;start_log_offset=124848;start_log_timestamp=1469311346327;start_log_lag=0;end_log_offset=124857;end_log_timestamp=1469311386135;end_log_lag=0;\r\n\r\n    2016/08/26 14:34:48 ;source_timestamp=2016-08-26T14:34:48Z;cluster=local;consumer_group=cd-changeevent-consumer;error=false;status=ERR;complete=true;lag_type=Partition;topic=directory-orgunits-change-feed;partition=0;partition_status=OK;start_log_offset=23;start_log_timestamp=1472219388000;start_log_lag=0;end_log_offset=23;end_log_timestamp=1472222088000;end_log_lag=0;\r\n\r\nWhy might this happen??\r\n\r\nThank you!!"},"comment":{"url":"https://api.github.com/repos/linkedin/Burrow/issues/comments/242861077","html_url":"https://github.com/linkedin/Burrow/issues/123#issuecomment-242861077","issue_url":"https://api.github.com/repos/linkedin/Burrow/issues/123","id":242861077,"user":{"login":"toddpalino","id":1283289,"avatar_url":"https://avatars.githubusercontent.com/u/1283289?v=3","gravatar_id":"","url":"https://api.github.com/users/toddpalino","html_url":"https://github.com/toddpalino","followers_url":"https://api.github.com/users/toddpalino/followers","following_url":"https://api.github.com/users/toddpalino/following{/other_user}","gists_url":"https://api.github.com/users/toddpalino/gists{/gist_id}","starred_url":"https://api.github.com/users/toddpalino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/toddpalino/subscriptions","organizations_url":"https://api.github.com/users/toddpalino/orgs","repos_url":"https://api.github.com/users/toddpalino/repos","events_url":"https://api.github.com/users/toddpalino/events{/privacy}","received_events_url":"https://api.github.com/users/toddpalino/received_events","type":"User","site_admin":false},"created_at":"2016-08-26T21:53:15Z","updated_at":"2016-08-26T21:53:15Z","body":"So those consumers are marked as ERR because they are stopped for at least one partition. That means that they have not committed an offset at all for an extended period of time. The lag for a consumer's partition is calculated on offset commit, so it's entirely possible that the current broker offset is ahead of the last committed offset, and therefore there is lag. This is a problem with the current method of calculating lag.\r\n\r\nAnd yes, if there is a problem with a single partition, then the consumer group is in a bad state."}}
{"actor":{"display_login":"toddpalino"}}