{"action":"created","issue":{"url":"https://api.github.com/repos/istio/istio/issues/8906","repository_url":"https://api.github.com/repos/istio/istio","labels_url":"https://api.github.com/repos/istio/istio/issues/8906/labels{/name}","comments_url":"https://api.github.com/repos/istio/istio/issues/8906/comments","events_url":"https://api.github.com/repos/istio/istio/issues/8906/events","html_url":"https://github.com/istio/istio/issues/8906","id":362748966,"node_id":"MDU6SXNzdWUzNjI3NDg5NjY=","number":8906,"title":"Telemetry: An error has occurred during metrics gathering: label dimensions inconsistent with previously collected metrics in the same metric family","user":{"login":"bobbytables","id":521627,"node_id":"MDQ6VXNlcjUyMTYyNw==","avatar_url":"https://avatars2.githubusercontent.com/u/521627?v=4","gravatar_id":"","url":"https://api.github.com/users/bobbytables","html_url":"https://github.com/bobbytables","followers_url":"https://api.github.com/users/bobbytables/followers","following_url":"https://api.github.com/users/bobbytables/following{/other_user}","gists_url":"https://api.github.com/users/bobbytables/gists{/gist_id}","starred_url":"https://api.github.com/users/bobbytables/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bobbytables/subscriptions","organizations_url":"https://api.github.com/users/bobbytables/orgs","repos_url":"https://api.github.com/users/bobbytables/repos","events_url":"https://api.github.com/users/bobbytables/events{/privacy}","received_events_url":"https://api.github.com/users/bobbytables/received_events","type":"User","site_admin":false},"labels":[{"id":813133538,"node_id":"MDU6TGFiZWw4MTMxMzM1Mzg=","url":"https://api.github.com/repos/istio/istio/labels/area/policies%20and%20telemetry","name":"area/policies and telemetry","color":"0e8a16","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":30,"created_at":"2018-09-21T19:38:32Z","updated_at":"2018-10-01T17:41:16Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"**Describe the bug**\r\nIstio Telemetry starts to fail on `/metrics` endpoint for prometheus metrics\r\n\r\n**Expected behavior**\r\nConsistent labels should exist\r\n\r\n**Steps to reproduce the bug**\r\nInstall istio in a system that emits metrics and wait I guess?\r\n\r\n**Version**\r\nKubernets 1.9 + Istio 1.0\r\n\r\n**Installation**\r\nHelm installation with slight modification\r\n\r\n**Environment**\r\nAWS w/ Kube-AWS\r\n\r\n**Cluster state**\r\nThe amount of data the tool exposes about our cluster is unacceptable for me to post here.\r\n\r\nFull error:\r\n\r\n```\r\nAn error has occurred during metrics gathering:\r\n\r\n2 error(s) occurred:\r\n* collected metric istio_request_bytes label:<name:\"connection_security_policy\" value:\"unknown\" > label:<name:\"destination_app\" value:\"permissions\" > label:<name:\"destination_principal\" value:\"unknown\" > label:<name:\"destination_service\" value:\"permissions-public.applications.svc.cluster.local\" > label:<name:\"destination_service_name\" value:\"permissions-public\" > label:<name:\"destination_service_namespace\" value:\"applications\" > label:<name:\"destination_version\" value:\"8\" > label:<name:\"destination_version\" value:\"8\" > label:<name:\"destination_workload\" value:\"permissions-grpc-v008\" > label:<name:\"reporter\" value:\"source\" > label:<name:\"request_protocol\" value:\"grpc\" > label:<name:\"response_code\" value:\"200\" > label:<name:\"source_app\" value:\"benefits\" > label:<name:\"source_principal\" value:\"unknown\" > label:<name:\"source_version\" value:\"unknown\" > label:<name:\"source_workload\" value:\"benefits-grpc\" > label:<name:\"source_workload_namespace\" value:\"applications\" > histogram:<sample_count:105 sample_sum:4515 bucket:<cumulative_count:0 upper_bound:1 > bucket:<cumulative_count:0 upper_bound:10 > bucket:<cumulative_count:105 upper_bound:100 > bucket:<cumulative_count:105 upper_bound:1000 > bucket:<cumulative_count:105 upper_bound:10000 > bucket:<cumulative_count:105 upper_bound:100000 > bucket:<cumulative_count:105 upper_bound:1e+06 > bucket:<cumulative_count:105 upper_bound:1e+07 > bucket:<cumulative_count:105 upper_bound:1e+08 > >  has label dimensions inconsistent with previously collected metrics in the same metric family\r\n* collected metric istio_request_bytes label:<name:\"connection_security_policy\" value:\"unknown\" > label:<name:\"destination_principal\" value:\"unknown\" > label:<name:\"destination_service\" value:\"sessions-grpc-gateway.applications.svc.cluster.local\" > label:<name:\"destination_service_name\" value:\"sessions-grpc-gateway\" > label:<name:\"destination_service_namespace\" value:\"applications\" > label:<name:\"destination_version\" value:\"unknown\" > label:<name:\"destination_version\" value:\"unknown\" > label:<name:\"destination_workload\" value:\"sessions\" > label:<name:\"destination_workload_namespace\" value:\"applications\" > label:<name:\"reporter\" value:\"source\" > label:<name:\"request_protocol\" value:\"http\" > label:<name:\"response_code\" value:\"200\" > label:<name:\"source_app\" value:\"unknown\" > label:<name:\"source_principal\" value:\"unknown\" > label:<name:\"source_version\" value:\"unknown\" > label:<name:\"source_workload\" value:\"sessions\" > label:<name:\"source_workload_namespace\" value:\"applications\" > histogram:<sample_count:29 sample_sum:275 bucket:<cumulative_count:28 upper_bound:1 > bucket:<cumulative_count:28 upper_bound:10 > bucket:<cumulative_count:28 upper_bound:100 > bucket:<cumulative_count:29 upper_bound:1000 > bucket:<cumulative_count:29 upper_bound:10000 > bucket:<cumulative_count:29 upper_bound:100000 > bucket:<cumulative_count:29 upper_bound:1e+06 > bucket:<cumulative_count:29 upper_bound:1e+07 > bucket:<cumulative_count:29 upper_bound:1e+08 > >  has label dimensions inconsistent with previously collected metrics in the same metric family\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/istio/istio/issues/comments/425997250","html_url":"https://github.com/istio/istio/issues/8906#issuecomment-425997250","issue_url":"https://api.github.com/repos/istio/istio/issues/8906","id":425997250,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNTk5NzI1MA==","user":{"login":"beorn7","id":5609886,"node_id":"MDQ6VXNlcjU2MDk4ODY=","avatar_url":"https://avatars3.githubusercontent.com/u/5609886?v=4","gravatar_id":"","url":"https://api.github.com/users/beorn7","html_url":"https://github.com/beorn7","followers_url":"https://api.github.com/users/beorn7/followers","following_url":"https://api.github.com/users/beorn7/following{/other_user}","gists_url":"https://api.github.com/users/beorn7/gists{/gist_id}","starred_url":"https://api.github.com/users/beorn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beorn7/subscriptions","organizations_url":"https://api.github.com/users/beorn7/orgs","repos_url":"https://api.github.com/users/beorn7/repos","events_url":"https://api.github.com/users/beorn7/events{/privacy}","received_events_url":"https://api.github.com/users/beorn7/received_events","type":"User","site_admin":false},"created_at":"2018-10-01T17:41:16Z","updated_at":"2018-10-01T17:41:16Z","author_association":"NONE","body":"I have just merged https://github.com/prometheus/client_golang/pull/474 . With that one in, you should get the correct error message if your case still occurs with the current state of client_golang."}}
{"actor":{"display_login":"beorn7"}}