{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/pushgateway/issues/214","repository_url":"https://api.github.com/repos/prometheus/pushgateway","labels_url":"https://api.github.com/repos/prometheus/pushgateway/issues/214/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/pushgateway/issues/214/comments","events_url":"https://api.github.com/repos/prometheus/pushgateway/issues/214/events","html_url":"https://github.com/prometheus/pushgateway/issues/214","id":369046672,"node_id":"MDU6SXNzdWUzNjkwNDY2NzI=","number":214,"title":"A simple way to make PushGateway always return 500","user":{"login":"wfhu","id":1288346,"node_id":"MDQ6VXNlcjEyODgzNDY=","avatar_url":"https://avatars0.githubusercontent.com/u/1288346?v=4","gravatar_id":"","url":"https://api.github.com/users/wfhu","html_url":"https://github.com/wfhu","followers_url":"https://api.github.com/users/wfhu/followers","following_url":"https://api.github.com/users/wfhu/following{/other_user}","gists_url":"https://api.github.com/users/wfhu/gists{/gist_id}","starred_url":"https://api.github.com/users/wfhu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wfhu/subscriptions","organizations_url":"https://api.github.com/users/wfhu/orgs","repos_url":"https://api.github.com/users/wfhu/repos","events_url":"https://api.github.com/users/wfhu/events{/privacy}","received_events_url":"https://api.github.com/users/wfhu/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-11T10:06:10Z","updated_at":"2018-10-11T10:44:58Z","closed_at":"2018-10-11T10:44:58Z","author_association":"NONE","body":"<!--\r\n\r\n\tNote that we use GitHub issues for bugs and (uncontroversial) feature\r\n\trequests (see below for details).\r\n\r\n\tPlease do *NOT* ask usage questions in GitHub issues.  Usage questions make\r\n\tmost sense on the users mailing list, where more people are available to\r\n\tpotentially respond to your question, and the whole community can benefit\r\n\tfrom the answers provided (perhaps your question has already been answered,\r\n\tsearch the archive to find out):\r\n\thttps://groups.google.com/forum/#!forum/prometheus-users\r\n\r\n\tWhile a GitHub issue is fine to track progress on an uncontroversial\r\n\tfeature request, many feature requests touch the best practices and\r\n\tconcepts of Prometheus as a whole and need to be discussed with the wider\r\n\tdeveloper community first. This is in particular true for a request to\r\n\treconsider a prior rejection of a feature request or any of the declared\r\n\tnon-goals (see README.md). Those overarching discussions happen on the\r\n\tdeveloper mailing list (GitHub issues, in particular closed ones, are not\r\n\ttracked by the wider developer community and thus inadequate):\r\n\thttps://groups.google.com/forum/#!forum/prometheus-developers\r\n\t\r\n\tYou can find more information at: https://prometheus.io/community/\r\n\r\n-->\r\n\r\n## Bug Report\r\n**What did you do?**\r\nWhen  I use shell to send redundent metrics to pushgateway service via HTTP request, the pushgateway become unusable, because it will always return 500 Internel  Server Error\r\n\r\nThe normal scripts looks like this:\r\n\r\n    cat <<EOF | curl --data-binary @-http://172.19.58.106:9091/metrics/job/huwf_job/instance/172.19.58.106\r\n    # TYPE some_metric counter\r\n    some_metric{label=\"val1\"} 42\r\n    # TYPE another_metric gauge\r\n    # HELP another_metric Just an example.\r\n    another_metric 2398.283\r\n\r\nThe above script is absolutely fine. But when I add one more line, and run it, the PushGateway will have problem.\r\n\r\nThe BAD scripts looks like this:\r\n\r\n    cat <<EOF | curl --data-binary @- http://172.19.58.106:9091/metrics/job/huwf_job/instance/172.19.58.106\r\n    # TYPE some_metric counter\r\n    some_metric{label=\"val1\"} 42\r\n    some_metric{label=\"val1\"} 42\r\n    # TYPE another_metric gauge\r\n    # HELP another_metric Just an example.\r\n    another_metric 2398.283\r\n\r\nPlease notice there are two same metrics-labels combination.\r\nAfter I run the BAD script, the pushgateway become unusable. I have to restart the pushgateway process.\r\n\r\n**What did you expect to see?**\r\nThis is like some kind of DoS attach, because anyone with access to the pushgateway service will be able to make it unusable. \r\nI think the pushgateway process should detect this kind of problem, and not become unusable. I think it would be fine to just report error to client and reject the data, or show a warning message and take the last pushed data.\r\n\r\n**What did you see instead? Under which circumstances?**\r\nThe pushgateway become ususable.\r\n\r\nThe error log of Prometheus:\r\n\r\n> level=debug ts=2018-10-11T08:20:55.640547371Z caller=scrape.go:687 component=\"scrape manager\" scrape_pool=pushgateway target=http://127.0.0.1:9091/metrics msg=\"Scrape failed\" err=\"server returned HTTP status 500 Internal Server Error\"\r\nlevel=debug ts=2018-10-11T08:21:25.631180199Z caller=scrape.go:687 component=\"scrape manager\" scrape_pool=pushgateway target=http://127.0.0.1:9091/metrics msg=\"Scrape failed\" err=\"server returned HTTP status 500 Internal Server Error\"\r\nlevel=debug ts=2018-10-11T08:21:55.625134532Z caller=scrape.go:687 component=\"scrape manager\" scrape_pool=pushgateway target=http://127.0.0.1:9091/metrics msg=\"Scrape failed\" err=\"server returned HTTP status 500 Internal Server Error\"\r\nlevel=debug ts=2018-10-11T08:22:25.615441221Z caller=scrape.go:687 component=\"scrape manager\" scrape_pool=pushgateway target=http://127.0.0.1:9091/metrics msg=\"Scrape failed\" err=\"server returned HTTP status 500 Internal Server Error\"\r\nlevel=debug ts=2018-10-11T08:22:55.618985952Z caller=scrape.go:687 component=\"scrape manager\" scrape_pool=pushgateway target=http://127.0.0.1:9091/metrics msg=\"Scrape failed\" err=\"server returned HTTP status 500 Internal Server Error\"\r\nlevel=debug ts=2018-10-11T08:23:25.613090781Z caller=scrape.go:687 component=\"scrape manager\" scrape_pool=pushgateway target=http://127.0.0.1:9091/metrics msg=\"Scrape failed\" err=\"server returned HTTP status 500 Internal Server Error\"\r\n\r\n\r\n**Environment**\r\n\r\n* System information:\r\n\r\nCentOS Linux release 7.5.1804 (Core)\r\nLinux 3.10.0-862.2.3.el7.x86_64 x86_64\r\n\r\n* Pushgateway version:\r\n\r\npushgateway, version 0.5.2 (branch: HEAD, revision: 231071b3e0a57b58df717bba5f4e5d88cac95389)\r\n  build user:       root@c95e387ff56a\r\n  build date:       20180615-16:11:56\r\n  go version:       go1.10.3\r\n\r\nprometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc)\r\n  build user:       root@149e5b3f0829\r\n  build date:       20180314-14:15:45\r\n  go version:       go1.10\r\n\r\n\r\n* Pushgateway command line:\r\n\r\n\t/opt/push/pushgateway/bin/pushgateway --web.listen-address=:9091 --log.level=debug\r\n\r\n\r\n* Logs:\r\n```\r\nThe pushgateway error log:\r\n\r\ntime=\"2018-10-11T16:17:11+08:00\" level=debug msg=\"Failed to parse text, text format parsing error in line 1: expected float as value, got \\\"\\\"\" source=\"push.go:108\"\r\ntime=\"2018-10-11T16:18:11+08:00\" level=debug msg=\"Failed to parse text, text format parsing error in line 1: expected float as value, got \\\"\\\"\" source=\"push.go:108\"\r\ntime=\"2018-10-11T16:19:12+08:00\" level=debug msg=\"Failed to parse text, text format parsing error in line 1: expected float as value, got \\\"\\\"\" source=\"push.go:108\"\r\ntime=\"2018-10-11T16:20:11+08:00\" level=debug msg=\"Failed to parse text, text format parsing error in line 1: expected float as value, got \\\"\\\"\" source=\"push.go:108\"\r\ntime=\"2018-10-11T16:21:11+08:00\" level=debug msg=\"Failed to parse text, text format parsing error in line 1: expected float as value, got \\\"\\\"\" source=\"push.go:108\"\r\ntime=\"2018-10-11T16:22:11+08:00\" level=debug msg=\"Failed to parse text, text format parsing error in line 1: expected float as value, got \\\"\\\"\" source=\"push.go:108\"\r\ntime=\"2018-10-11T16:23:11+08:00\" level=debug msg=\"Failed to parse text, text format parsing error in line 1: expected float as value, got \\\"\\\"\" source=\"push.go:108\"\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/pushgateway/issues/comments/428909038","html_url":"https://github.com/prometheus/pushgateway/issues/214#issuecomment-428909038","issue_url":"https://api.github.com/repos/prometheus/pushgateway/issues/214","id":428909038,"node_id":"MDEyOklzc3VlQ29tbWVudDQyODkwOTAzOA==","user":{"login":"beorn7","id":5609886,"node_id":"MDQ6VXNlcjU2MDk4ODY=","avatar_url":"https://avatars3.githubusercontent.com/u/5609886?v=4","gravatar_id":"","url":"https://api.github.com/users/beorn7","html_url":"https://github.com/beorn7","followers_url":"https://api.github.com/users/beorn7/followers","following_url":"https://api.github.com/users/beorn7/following{/other_user}","gists_url":"https://api.github.com/users/beorn7/gists{/gist_id}","starred_url":"https://api.github.com/users/beorn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beorn7/subscriptions","organizations_url":"https://api.github.com/users/beorn7/orgs","repos_url":"https://api.github.com/users/beorn7/repos","events_url":"https://api.github.com/users/beorn7/events{/privacy}","received_events_url":"https://api.github.com/users/beorn7/received_events","type":"User","site_admin":false},"created_at":"2018-10-11T10:44:58Z","updated_at":"2018-10-11T10:44:58Z","author_association":"MEMBER","body":"This is a dupe of https://github.com/prometheus/pushgateway/issues/202"}}
{"actor":{"display_login":"beorn7"}}