{"action":"created","comment":{"url":"https://api.github.com/repos/alphagov/paas-cf/pulls/comments/241041659","pull_request_review_id":184212636,"id":241041659,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI0MTA0MTY1OQ==","diff_hunk":"@@ -0,0 +1,16 @@\n+# Source: concourse\n+---\n+\n+- type: replace\n+  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties/prometheus/custom_rules?/-\n+  value:\n+    name: ConcourseSmoketestsNoData\n+    rules:\n+      - alert: ConcourseSmoketestsNoData\n+        expr: absent(concourse_builds_finished{pipeline=\"create-cloudfoundry\",exported_job=\"continuous-smoke-tests\"})","path":"manifests/prometheus/alerts.d/concourse-smoketests-no-data.yml","position":10,"original_position":10,"commit_id":"c1e99394ff05322ed09f0a129c4393d563f62782","original_commit_id":"c1e99394ff05322ed09f0a129c4393d563f62782","user":{"login":"jpluscplusm","id":224508,"node_id":"MDQ6VXNlcjIyNDUwOA==","avatar_url":"https://avatars0.githubusercontent.com/u/224508?v=4","gravatar_id":"","url":"https://api.github.com/users/jpluscplusm","html_url":"https://github.com/jpluscplusm","followers_url":"https://api.github.com/users/jpluscplusm/followers","following_url":"https://api.github.com/users/jpluscplusm/following{/other_user}","gists_url":"https://api.github.com/users/jpluscplusm/gists{/gist_id}","starred_url":"https://api.github.com/users/jpluscplusm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpluscplusm/subscriptions","organizations_url":"https://api.github.com/users/jpluscplusm/orgs","repos_url":"https://api.github.com/users/jpluscplusm/repos","events_url":"https://api.github.com/users/jpluscplusm/events{/privacy}","received_events_url":"https://api.github.com/users/jpluscplusm/received_events","type":"User","site_admin":false},"body":"s/exported_job/job/g","created_at":"2018-12-12T14:45:45Z","updated_at":"2018-12-12T14:45:45Z","html_url":"https://github.com/alphagov/paas-cf/pull/1681#discussion_r241041659","pull_request_url":"https://api.github.com/repos/alphagov/paas-cf/pulls/1681","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/alphagov/paas-cf/pulls/comments/241041659"},"html":{"href":"https://github.com/alphagov/paas-cf/pull/1681#discussion_r241041659"},"pull_request":{"href":"https://api.github.com/repos/alphagov/paas-cf/pulls/1681"}}},"pull_request":{"url":"https://api.github.com/repos/alphagov/paas-cf/pulls/1681","id":238060403,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM4MDYwNDAz","html_url":"https://github.com/alphagov/paas-cf/pull/1681","diff_url":"https://github.com/alphagov/paas-cf/pull/1681.diff","patch_url":"https://github.com/alphagov/paas-cf/pull/1681.patch","issue_url":"https://api.github.com/repos/alphagov/paas-cf/issues/1681","number":1681,"state":"open","locked":false,"title":"[#162448881] Enable scraping Concourse metrics in Prometheus, add related missing alerts","user":{"login":"bandesz","id":868104,"node_id":"MDQ6VXNlcjg2ODEwNA==","avatar_url":"https://avatars0.githubusercontent.com/u/868104?v=4","gravatar_id":"","url":"https://api.github.com/users/bandesz","html_url":"https://github.com/bandesz","followers_url":"https://api.github.com/users/bandesz/followers","following_url":"https://api.github.com/users/bandesz/following{/other_user}","gists_url":"https://api.github.com/users/bandesz/gists{/gist_id}","starred_url":"https://api.github.com/users/bandesz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bandesz/subscriptions","organizations_url":"https://api.github.com/users/bandesz/orgs","repos_url":"https://api.github.com/users/bandesz/repos","events_url":"https://api.github.com/users/bandesz/events{/privacy}","received_events_url":"https://api.github.com/users/bandesz/received_events","type":"User","site_admin":false},"body":"# What\r\n\r\nReenable the Prometheus scraping for Concourse and migrate the continuous-smoke-tests and check-certificate monitors from DataDog.\r\n\r\nI decided to record a more complex expression like:\r\n```\r\nsum(\r\n  concourse_builds_finished{exported_job=\"check-certificates\",pipeline=\"create-cloudfoundry\",status=\"errored\"}\r\n  or\r\n  (absent(concourse_builds_finished{exported_job=\"check-certificates\",pipeline=\"create-cloudfoundry\",status=\"errored\"})-1)\r\n)\r\n```\r\nas this way we'll have 0 values if Concourse doesn't report anything for a specific metric (it starts to report one after the first event only). This way we can consistently check the increase of any error/failure rate.\r\n\r\n# How to review\r\n\r\n1. Review and deploy https://github.com/alphagov/paas-bootstrap/pull/227 first\r\n1. Deploy your CF from this branch\r\n1. Check if Prometheus started scraping the Concourse metrics\r\n1. You can verify the alerts by manually making the continuous-smoke-tests/check-certificates to fail/error (add an `exit 1` to any of these jobs and run the `pipelines` target)\r\n\r\n# Who can review\r\n\r\nNot me","created_at":"2018-12-12T14:39:29Z","updated_at":"2018-12-12T14:45:45Z","closed_at":null,"merged_at":null,"merge_commit_sha":"1c517e23a0a01ddf9909d359fa667366caead39a","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/alphagov/paas-cf/pulls/1681/commits","review_comments_url":"https://api.github.com/repos/alphagov/paas-cf/pulls/1681/comments","review_comment_url":"https://api.github.com/repos/alphagov/paas-cf/pulls/comments{/number}","comments_url":"https://api.github.com/repos/alphagov/paas-cf/issues/1681/comments","statuses_url":"https://api.github.com/repos/alphagov/paas-cf/statuses/c1e99394ff05322ed09f0a129c4393d563f62782","head":{"label":"alphagov:prometheus_concourse_162448881","ref":"prometheus_concourse_162448881","sha":"c1e99394ff05322ed09f0a129c4393d563f62782","user":{"login":"alphagov","id":596977,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5Njk3Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/596977?v=4","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"repo":{"id":46716359,"node_id":"MDEwOlJlcG9zaXRvcnk0NjcxNjM1OQ==","name":"paas-cf","full_name":"alphagov/paas-cf","private":false,"owner":{"login":"alphagov","id":596977,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5Njk3Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/596977?v=4","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/alphagov/paas-cf","description":"Government PaaS - Cloud Foundry","fork":false,"url":"https://api.github.com/repos/alphagov/paas-cf","forks_url":"https://api.github.com/repos/alphagov/paas-cf/forks","keys_url":"https://api.github.com/repos/alphagov/paas-cf/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alphagov/paas-cf/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alphagov/paas-cf/teams","hooks_url":"https://api.github.com/repos/alphagov/paas-cf/hooks","issue_events_url":"https://api.github.com/repos/alphagov/paas-cf/issues/events{/number}","events_url":"https://api.github.com/repos/alphagov/paas-cf/events","assignees_url":"https://api.github.com/repos/alphagov/paas-cf/assignees{/user}","branches_url":"https://api.github.com/repos/alphagov/paas-cf/branches{/branch}","tags_url":"https://api.github.com/repos/alphagov/paas-cf/tags","blobs_url":"https://api.github.com/repos/alphagov/paas-cf/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alphagov/paas-cf/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alphagov/paas-cf/git/refs{/sha}","trees_url":"https://api.github.com/repos/alphagov/paas-cf/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alphagov/paas-cf/statuses/{sha}","languages_url":"https://api.github.com/repos/alphagov/paas-cf/languages","stargazers_url":"https://api.github.com/repos/alphagov/paas-cf/stargazers","contributors_url":"https://api.github.com/repos/alphagov/paas-cf/contributors","subscribers_url":"https://api.github.com/repos/alphagov/paas-cf/subscribers","subscription_url":"https://api.github.com/repos/alphagov/paas-cf/subscription","commits_url":"https://api.github.com/repos/alphagov/paas-cf/commits{/sha}","git_commits_url":"https://api.github.com/repos/alphagov/paas-cf/git/commits{/sha}","comments_url":"https://api.github.com/repos/alphagov/paas-cf/comments{/number}","issue_comment_url":"https://api.github.com/repos/alphagov/paas-cf/issues/comments{/number}","contents_url":"https://api.github.com/repos/alphagov/paas-cf/contents/{+path}","compare_url":"https://api.github.com/repos/alphagov/paas-cf/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alphagov/paas-cf/merges","archive_url":"https://api.github.com/repos/alphagov/paas-cf/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alphagov/paas-cf/downloads","issues_url":"https://api.github.com/repos/alphagov/paas-cf/issues{/number}","pulls_url":"https://api.github.com/repos/alphagov/paas-cf/pulls{/number}","milestones_url":"https://api.github.com/repos/alphagov/paas-cf/milestones{/number}","notifications_url":"https://api.github.com/repos/alphagov/paas-cf/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alphagov/paas-cf/labels{/name}","releases_url":"https://api.github.com/repos/alphagov/paas-cf/releases{/id}","deployments_url":"https://api.github.com/repos/alphagov/paas-cf/deployments","created_at":"2015-11-23T11:36:57Z","updated_at":"2018-12-11T16:22:47Z","pushed_at":"2018-12-12T14:39:30Z","git_url":"git://github.com/alphagov/paas-cf.git","ssh_url":"git@github.com:alphagov/paas-cf.git","clone_url":"https://github.com/alphagov/paas-cf.git","svn_url":"https://github.com/alphagov/paas-cf","homepage":null,"size":50762,"stargazers_count":43,"watchers_count":43,"language":"HTML","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":12,"mirror_url":null,"archived":false,"open_issues_count":2,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":12,"open_issues":2,"watchers":43,"default_branch":"master"}},"base":{"label":"alphagov:master","ref":"master","sha":"0856bfd1cb73097505bf3ff1640fd3bcb23c7890","user":{"login":"alphagov","id":596977,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5Njk3Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/596977?v=4","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"repo":{"id":46716359,"node_id":"MDEwOlJlcG9zaXRvcnk0NjcxNjM1OQ==","name":"paas-cf","full_name":"alphagov/paas-cf","private":false,"owner":{"login":"alphagov","id":596977,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5Njk3Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/596977?v=4","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/alphagov/paas-cf","description":"Government PaaS - Cloud Foundry","fork":false,"url":"https://api.github.com/repos/alphagov/paas-cf","forks_url":"https://api.github.com/repos/alphagov/paas-cf/forks","keys_url":"https://api.github.com/repos/alphagov/paas-cf/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alphagov/paas-cf/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alphagov/paas-cf/teams","hooks_url":"https://api.github.com/repos/alphagov/paas-cf/hooks","issue_events_url":"https://api.github.com/repos/alphagov/paas-cf/issues/events{/number}","events_url":"https://api.github.com/repos/alphagov/paas-cf/events","assignees_url":"https://api.github.com/repos/alphagov/paas-cf/assignees{/user}","branches_url":"https://api.github.com/repos/alphagov/paas-cf/branches{/branch}","tags_url":"https://api.github.com/repos/alphagov/paas-cf/tags","blobs_url":"https://api.github.com/repos/alphagov/paas-cf/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alphagov/paas-cf/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alphagov/paas-cf/git/refs{/sha}","trees_url":"https://api.github.com/repos/alphagov/paas-cf/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alphagov/paas-cf/statuses/{sha}","languages_url":"https://api.github.com/repos/alphagov/paas-cf/languages","stargazers_url":"https://api.github.com/repos/alphagov/paas-cf/stargazers","contributors_url":"https://api.github.com/repos/alphagov/paas-cf/contributors","subscribers_url":"https://api.github.com/repos/alphagov/paas-cf/subscribers","subscription_url":"https://api.github.com/repos/alphagov/paas-cf/subscription","commits_url":"https://api.github.com/repos/alphagov/paas-cf/commits{/sha}","git_commits_url":"https://api.github.com/repos/alphagov/paas-cf/git/commits{/sha}","comments_url":"https://api.github.com/repos/alphagov/paas-cf/comments{/number}","issue_comment_url":"https://api.github.com/repos/alphagov/paas-cf/issues/comments{/number}","contents_url":"https://api.github.com/repos/alphagov/paas-cf/contents/{+path}","compare_url":"https://api.github.com/repos/alphagov/paas-cf/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alphagov/paas-cf/merges","archive_url":"https://api.github.com/repos/alphagov/paas-cf/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alphagov/paas-cf/downloads","issues_url":"https://api.github.com/repos/alphagov/paas-cf/issues{/number}","pulls_url":"https://api.github.com/repos/alphagov/paas-cf/pulls{/number}","milestones_url":"https://api.github.com/repos/alphagov/paas-cf/milestones{/number}","notifications_url":"https://api.github.com/repos/alphagov/paas-cf/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alphagov/paas-cf/labels{/name}","releases_url":"https://api.github.com/repos/alphagov/paas-cf/releases{/id}","deployments_url":"https://api.github.com/repos/alphagov/paas-cf/deployments","created_at":"2015-11-23T11:36:57Z","updated_at":"2018-12-11T16:22:47Z","pushed_at":"2018-12-12T14:39:30Z","git_url":"git://github.com/alphagov/paas-cf.git","ssh_url":"git@github.com:alphagov/paas-cf.git","clone_url":"https://github.com/alphagov/paas-cf.git","svn_url":"https://github.com/alphagov/paas-cf","homepage":null,"size":50762,"stargazers_count":43,"watchers_count":43,"language":"HTML","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":12,"mirror_url":null,"archived":false,"open_issues_count":2,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":12,"open_issues":2,"watchers":43,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/alphagov/paas-cf/pulls/1681"},"html":{"href":"https://github.com/alphagov/paas-cf/pull/1681"},"issue":{"href":"https://api.github.com/repos/alphagov/paas-cf/issues/1681"},"comments":{"href":"https://api.github.com/repos/alphagov/paas-cf/issues/1681/comments"},"review_comments":{"href":"https://api.github.com/repos/alphagov/paas-cf/pulls/1681/comments"},"review_comment":{"href":"https://api.github.com/repos/alphagov/paas-cf/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/alphagov/paas-cf/pulls/1681/commits"},"statuses":{"href":"https://api.github.com/repos/alphagov/paas-cf/statuses/c1e99394ff05322ed09f0a129c4393d563f62782"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"jpluscplusm"}}