{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla/kitsune/issues/3041","repository_url":"https://api.github.com/repos/mozilla/kitsune","labels_url":"https://api.github.com/repos/mozilla/kitsune/issues/3041/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/kitsune/issues/3041/comments","events_url":"https://api.github.com/repos/mozilla/kitsune/issues/3041/events","html_url":"https://github.com/mozilla/kitsune/issues/3041","id":293905230,"node_id":"MDU6SXNzdWUyOTM5MDUyMzA=","number":3041,"title":"Setup Datadog Monitor for `celery.lag` metric","user":{"login":"glogiotatidis","id":584352,"node_id":"MDQ6VXNlcjU4NDM1Mg==","avatar_url":"https://avatars3.githubusercontent.com/u/584352?v=4","gravatar_id":"","url":"https://api.github.com/users/glogiotatidis","html_url":"https://github.com/glogiotatidis","followers_url":"https://api.github.com/users/glogiotatidis/followers","following_url":"https://api.github.com/users/glogiotatidis/following{/other_user}","gists_url":"https://api.github.com/users/glogiotatidis/gists{/gist_id}","starred_url":"https://api.github.com/users/glogiotatidis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glogiotatidis/subscriptions","organizations_url":"https://api.github.com/users/glogiotatidis/orgs","repos_url":"https://api.github.com/users/glogiotatidis/repos","events_url":"https://api.github.com/users/glogiotatidis/events{/privacy}","received_events_url":"https://api.github.com/users/glogiotatidis/received_events","type":"User","site_admin":false},"labels":[{"id":898203068,"node_id":"MDU6TGFiZWw4OTgyMDMwNjg=","url":"https://api.github.com/repos/mozilla/kitsune/labels/Now","name":"Now","color":"ddea88","default":false}],"state":"open","locked":false,"assignee":{"login":"glogiotatidis","id":584352,"node_id":"MDQ6VXNlcjU4NDM1Mg==","avatar_url":"https://avatars3.githubusercontent.com/u/584352?v=4","gravatar_id":"","url":"https://api.github.com/users/glogiotatidis","html_url":"https://github.com/glogiotatidis","followers_url":"https://api.github.com/users/glogiotatidis/followers","following_url":"https://api.github.com/users/glogiotatidis/following{/other_user}","gists_url":"https://api.github.com/users/glogiotatidis/gists{/gist_id}","starred_url":"https://api.github.com/users/glogiotatidis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glogiotatidis/subscriptions","organizations_url":"https://api.github.com/users/glogiotatidis/orgs","repos_url":"https://api.github.com/users/glogiotatidis/repos","events_url":"https://api.github.com/users/glogiotatidis/events{/privacy}","received_events_url":"https://api.github.com/users/glogiotatidis/received_events","type":"User","site_admin":false},"assignees":[{"login":"glogiotatidis","id":584352,"node_id":"MDQ6VXNlcjU4NDM1Mg==","avatar_url":"https://avatars3.githubusercontent.com/u/584352?v=4","gravatar_id":"","url":"https://api.github.com/users/glogiotatidis","html_url":"https://github.com/glogiotatidis","followers_url":"https://api.github.com/users/glogiotatidis/followers","following_url":"https://api.github.com/users/glogiotatidis/following{/other_user}","gists_url":"https://api.github.com/users/glogiotatidis/gists{/gist_id}","starred_url":"https://api.github.com/users/glogiotatidis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glogiotatidis/subscriptions","organizations_url":"https://api.github.com/users/glogiotatidis/orgs","repos_url":"https://api.github.com/users/glogiotatidis/repos","events_url":"https://api.github.com/users/glogiotatidis/events{/privacy}","received_events_url":"https://api.github.com/users/glogiotatidis/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2018-02-02T14:26:45Z","updated_at":"2018-07-23T12:22:34Z","closed_at":null,"author_association":"MEMBER","body":"Part of #3015"},"comment":{"url":"https://api.github.com/repos/mozilla/kitsune/issues/comments/407039425","html_url":"https://github.com/mozilla/kitsune/issues/3041#issuecomment-407039425","issue_url":"https://api.github.com/repos/mozilla/kitsune/issues/3041","id":407039425,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNzAzOTQyNQ==","user":{"login":"glogiotatidis","id":584352,"node_id":"MDQ6VXNlcjU4NDM1Mg==","avatar_url":"https://avatars3.githubusercontent.com/u/584352?v=4","gravatar_id":"","url":"https://api.github.com/users/glogiotatidis","html_url":"https://github.com/glogiotatidis","followers_url":"https://api.github.com/users/glogiotatidis/followers","following_url":"https://api.github.com/users/glogiotatidis/following{/other_user}","gists_url":"https://api.github.com/users/glogiotatidis/gists{/gist_id}","starred_url":"https://api.github.com/users/glogiotatidis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glogiotatidis/subscriptions","organizations_url":"https://api.github.com/users/glogiotatidis/orgs","repos_url":"https://api.github.com/users/glogiotatidis/repos","events_url":"https://api.github.com/users/glogiotatidis/events{/privacy}","received_events_url":"https://api.github.com/users/glogiotatidis/received_events","type":"User","site_admin":false},"created_at":"2018-07-23T12:22:34Z","updated_at":"2018-07-23T12:22:34Z","author_association":"MEMBER","body":"After talking with Datadog support we found a way to combine the two metrics (*) into one Monitor:\r\n\r\n`default(max:sumo_prod.oregon_a.celery.lag{*}, 0) + default(max:sumo_prod.oregon_b.celery.lag{*}, 0)`\r\n\r\nThis combines the two metrics with addition and uses default value of `0` when there's no value. The reason we're using default is that due to the nature of the metric: The task it's either executed by a worker `oregon_a` or a worker in `oregon_b`. This means that when `oregon_a` reports, `oregon_b` doesn't and the other way around. By using a default value we can make this Monitor work and not complain about missing data.\r\n\r\nI added useful information on how to restart the celery workers in the metric notification email. I'll continue to monitor the configuration for one week before adding the team to the recipients list to avoid false positives.\r\n\r\n(*) Celery lag measurement happens with a Celery task. Since we're running celery workers on two different clusters we have `snippets_prod.oregon_a.celery.lag` and `snippets_prod.oregon_b.celery.lag` which both have to be taken into account to eval celery lag."}}
{"actor":{"display_login":"glogiotatidis"}}