{"action":"created","issue":{"url":"https://api.github.com/repos/concourse/concourse/issues/2244","repository_url":"https://api.github.com/repos/concourse/concourse","labels_url":"https://api.github.com/repos/concourse/concourse/issues/2244/labels{/name}","comments_url":"https://api.github.com/repos/concourse/concourse/issues/2244/comments","events_url":"https://api.github.com/repos/concourse/concourse/issues/2244/events","html_url":"https://github.com/concourse/concourse/issues/2244","id":327747039,"node_id":"MDU6SXNzdWUzMjc3NDcwMzk=","number":2244,"title":"Prometheus metrics include dead workers","user":{"login":"jmcarp","id":1633460,"node_id":"MDQ6VXNlcjE2MzM0NjA=","avatar_url":"https://avatars0.githubusercontent.com/u/1633460?v=4","gravatar_id":"","url":"https://api.github.com/users/jmcarp","html_url":"https://github.com/jmcarp","followers_url":"https://api.github.com/users/jmcarp/followers","following_url":"https://api.github.com/users/jmcarp/following{/other_user}","gists_url":"https://api.github.com/users/jmcarp/gists{/gist_id}","starred_url":"https://api.github.com/users/jmcarp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmcarp/subscriptions","organizations_url":"https://api.github.com/users/jmcarp/orgs","repos_url":"https://api.github.com/users/jmcarp/repos","events_url":"https://api.github.com/users/jmcarp/events{/privacy}","received_events_url":"https://api.github.com/users/jmcarp/received_events","type":"User","site_admin":false},"labels":[{"id":764529113,"node_id":"MDU6TGFiZWw3NjQ1MjkxMTM=","url":"https://api.github.com/repos/concourse/concourse/labels/accepted","name":"accepted","color":"28a745","default":false},{"id":93695076,"node_id":"MDU6TGFiZWw5MzY5NTA3Ng==","url":"https://api.github.com/repos/concourse/concourse/labels/bug","name":"bug","color":"d73a4a","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/concourse/concourse/milestones/32","html_url":"https://github.com/concourse/concourse/milestone/32","labels_url":"https://api.github.com/repos/concourse/concourse/milestones/32/labels","id":3460090,"node_id":"MDk6TWlsZXN0b25lMzQ2MDA5MA==","number":32,"title":"v4.0.0","description":null,"creator":{"login":"vito","id":1880,"node_id":"MDQ6VXNlcjE4ODA=","avatar_url":"https://avatars1.githubusercontent.com/u/1880?v=4","gravatar_id":"","url":"https://api.github.com/users/vito","html_url":"https://github.com/vito","followers_url":"https://api.github.com/users/vito/followers","following_url":"https://api.github.com/users/vito/following{/other_user}","gists_url":"https://api.github.com/users/vito/gists{/gist_id}","starred_url":"https://api.github.com/users/vito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vito/subscriptions","organizations_url":"https://api.github.com/users/vito/orgs","repos_url":"https://api.github.com/users/vito/repos","events_url":"https://api.github.com/users/vito/events{/privacy}","received_events_url":"https://api.github.com/users/vito/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":50,"state":"closed","created_at":"2018-06-28T15:56:15Z","updated_at":"2018-07-30T17:59:08Z","due_on":null,"closed_at":"2018-07-30T14:06:04Z"},"comments":6,"created_at":"2018-05-30T14:05:44Z","updated_at":"2018-08-24T14:36:22Z","closed_at":"2018-07-24T14:17:32Z","author_association":"CONTRIBUTOR","body":"# Bug Report\r\n\r\nThe prometheus metrics emitter never resets or deletes metrics for dead workers, so the prometheus endpoint reports container counts, volume counts, etc. for workers long after they're retired. We noticed this because one of our workers had a high container count right as it was retired, and we alert on high container counts, so we saw a phantom alert firing even though the worker it referenced didn't exist.\r\n\r\nNot sure how to fix this with the current metrics interface, since the prometheus emitter doesn't know when workers get garbage collected.\r\n\r\ncc @TimSimmons @databus23 \r\n\r\nThe following can also be handy:\r\n\r\n* Concourse version: 3.13.0\r\n* Deployment type (BOSH/Docker/binary): BOSH\r\n* Infrastructure/IaaS: AWS\r\n* Browser (if applicable): n/a\r\n* Did this used to work? n/a\r\n"},"comment":{"url":"https://api.github.com/repos/concourse/concourse/issues/comments/415778544","html_url":"https://github.com/concourse/concourse/issues/2244#issuecomment-415778544","issue_url":"https://api.github.com/repos/concourse/concourse/issues/2244","id":415778544,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNTc3ODU0NA==","user":{"login":"marco-m","id":3075069,"node_id":"MDQ6VXNlcjMwNzUwNjk=","avatar_url":"https://avatars3.githubusercontent.com/u/3075069?v=4","gravatar_id":"","url":"https://api.github.com/users/marco-m","html_url":"https://github.com/marco-m","followers_url":"https://api.github.com/users/marco-m/followers","following_url":"https://api.github.com/users/marco-m/following{/other_user}","gists_url":"https://api.github.com/users/marco-m/gists{/gist_id}","starred_url":"https://api.github.com/users/marco-m/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marco-m/subscriptions","organizations_url":"https://api.github.com/users/marco-m/orgs","repos_url":"https://api.github.com/users/marco-m/repos","events_url":"https://api.github.com/users/marco-m/events{/privacy}","received_events_url":"https://api.github.com/users/marco-m/received_events","type":"User","site_admin":false},"created_at":"2018-08-24T14:36:22Z","updated_at":"2018-08-24T14:36:22Z","author_association":"NONE","body":"Adding a comment to this closed bug because it seems to me that this thread contains good discussion about what Prometheus expects and the difficulty in making it work for Concourse.\r\n\r\nI found https://www.robustperception.io/its-easy-to-convert-pull-to-push which explains the problems in adapting to pull (that is, to Prometheus) a system designed to push (as Concourse). They explain that on the other hand, if a system is designed for pulling, then it is easier to adapt it to pushing. This (although I understand would be a major undertaking) looks a promising path.\r\n\r\nOpinions ?"}}
{"actor":{"display_login":"marco-m"}}