{"action":"created","issue":{"url":"https://api.github.com/repos/department-of-veterans-affairs/caseflow-efolder/issues/987","repository_url":"https://api.github.com/repos/department-of-veterans-affairs/caseflow-efolder","labels_url":"https://api.github.com/repos/department-of-veterans-affairs/caseflow-efolder/issues/987/labels{/name}","comments_url":"https://api.github.com/repos/department-of-veterans-affairs/caseflow-efolder/issues/987/comments","events_url":"https://api.github.com/repos/department-of-veterans-affairs/caseflow-efolder/issues/987/events","html_url":"https://github.com/department-of-veterans-affairs/caseflow-efolder/issues/987","id":315297152,"number":987,"title":"`/api/v2/records/:uuid` is generating a massive number of data series in Prometheus","user":{"login":"askldjd","id":3258724,"avatar_url":"https://avatars0.githubusercontent.com/u/3258724?v=4","gravatar_id":"","url":"https://api.github.com/users/askldjd","html_url":"https://github.com/askldjd","followers_url":"https://api.github.com/users/askldjd/followers","following_url":"https://api.github.com/users/askldjd/following{/other_user}","gists_url":"https://api.github.com/users/askldjd/gists{/gist_id}","starred_url":"https://api.github.com/users/askldjd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/askldjd/subscriptions","organizations_url":"https://api.github.com/users/askldjd/orgs","repos_url":"https://api.github.com/users/askldjd/repos","events_url":"https://api.github.com/users/askldjd/events{/privacy}","received_events_url":"https://api.github.com/users/askldjd/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-18T01:31:43Z","updated_at":"2018-04-18T02:01:36Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"The root cause is https://github.com/department-of-veterans-affairs/appeals-deployment/pull/1377. Since this feature was turned on, Prometheus has been crashing. At its peak, we observed over 250k series per `/metrics` API call. Since Prometheus polls `/metrics` every 15s, it was not able handle the load.\r\n\r\nTo fix this, we need to implement a label collapser in eX similar to one in Caseflow.\r\nhttps://github.com/department-of-veterans-affairs/caseflow/blob/master/config.ru#L25\r\n\r\nHere's a sample `/metric` output from efolder server.\r\n```\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"0.005\"} 1.0\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"0.01\"} 1.0\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"0.025\"} 1.0\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"0.05\"} 1.0\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"0.1\"} 1.0\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"0.25\"} 1.0\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"0.5\"} 1.0\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"1\"} 1.0\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"2.5\"} 1.0\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"5\"} 1.0\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"10\"} 1.0\r\nhttp_server_request_duration_seconds_bucket{code=\"200\",method=\"options\",host=\"efolder.cf.ds.va.gov\",path=\"/api/v2/records/7D11AE16-1DFF-49F5-AE31-B9E4675EBC30\",le=\"+Inf\"} 1.0\r\n```\r\n\r\nTo overcome this, I hacked the `/etc/prometheus/prometheus.yml` file to skip over all efolder servers.\r\n\r\n```yml\r\n  - job_name: 'rails-metrics'\r\n    ec2_sd_configs:\r\n      - region: us-gov-west-1\r\n        port: 3000\r\n    relabel_configs:\r\n        - source_labels: [__meta_ec2_tag_deploy_env]\r\n          regex: prod\r\n          action: keep\r\n        - source_labels: [__meta_ec2_instance_id]\r\n          target_label: instance\r\n        - source_labels: [__meta_ec2_tag_app_name]\r\n          target_label: appname\r\n        - source_labels: [__meta_ec2_tag_app_name]   <- drop action on all efolder servers\r\n          regex: efolder\r\n          action: drop\r\n        - source_labels: [__meta_ec2_tag_deploy_env]\r\n          target_label: env\r\n        - source_labels: [__meta_ec2_tag_type]\r\n          regex: utility\r\n          action: drop\r\n```\r\n\r\n@lisac-usds \r\n@RobertAlexanderUSDS \r\n@sunil-sadasivan \r\n@lowellrex \r\n@mdbenjam "},"comment":{"url":"https://api.github.com/repos/department-of-veterans-affairs/caseflow-efolder/issues/comments/382221522","html_url":"https://github.com/department-of-veterans-affairs/caseflow-efolder/issues/987#issuecomment-382221522","issue_url":"https://api.github.com/repos/department-of-veterans-affairs/caseflow-efolder/issues/987","id":382221522,"user":{"login":"lisac-usds","id":31673962,"avatar_url":"https://avatars3.githubusercontent.com/u/31673962?v=4","gravatar_id":"","url":"https://api.github.com/users/lisac-usds","html_url":"https://github.com/lisac-usds","followers_url":"https://api.github.com/users/lisac-usds/followers","following_url":"https://api.github.com/users/lisac-usds/following{/other_user}","gists_url":"https://api.github.com/users/lisac-usds/gists{/gist_id}","starred_url":"https://api.github.com/users/lisac-usds/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lisac-usds/subscriptions","organizations_url":"https://api.github.com/users/lisac-usds/orgs","repos_url":"https://api.github.com/users/lisac-usds/repos","events_url":"https://api.github.com/users/lisac-usds/events{/privacy}","received_events_url":"https://api.github.com/users/lisac-usds/received_events","type":"User","site_admin":false},"created_at":"2018-04-18T02:01:35Z","updated_at":"2018-04-18T02:01:35Z","author_association":"NONE","body":"putting down some initial thoughts - what if we edit this snippet in [config.ru#L27](https://github.com/department-of-veterans-affairs/caseflow-efolder/blob/628fb30e53ef2e1ef23bcb4182ccc35ba9bc777b/config.ru#L37 ):\r\n```\r\nlabel_builder = lambda do |env, code|\r\n  {\r\n    code: code,\r\n    method: env[\"REQUEST_METHOD\"].downcase,\r\n    host: env[\"HTTP_HOST\"].to_s,\r\n    # Replace ids and id-like values to keep cardinality low.\r\n    # Otherwise Prometheus crashes on 400k+ data series.\r\n    # '/users/1234/comments' -> '/users/:id/comments'\r\n    # '/hearings/dockets/2017-10-15' -> '/hearings/dockets/:id'\r\n    # '/certifications/new/123C' -> '/certifications/new/:id'\r\n    path: env[\"PATH_INFO\"].to_s.gsub(%r{\\/\\d+-?\\d*-?\\d*C?S?(\\/|$)}, '/:id\\\\1')\r\n  }\r\nend\r\n```\r\n\r\nto also run a uid substitution on the path, ie:\r\n```\r\npath: env[\"PATH_INFO\"].to_s.gsub(%r{\\/\\d+-?\\d*-?\\d*C?S?(\\/|$)}, '/:id\\\\1').gsub(%r{\\/[A-F0-9]{8}-[A-F0-9]{4}-[A-F0-9]{4}-[A-F0-9]{4}-[A-F0-9]{12}(\\/|$)}, '/:id\\\\1')\r\n```"}}
{"actor":{"display_login":"lisac-usds"}}