{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/3886","repository_url":"https://api.github.com/repos/prometheus/prometheus","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/3886/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/3886/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/3886/events","html_url":"https://github.com/prometheus/prometheus/issues/3886","id":299986936,"number":3886,"title":"All data before the first scrape is lost when using rate / irate / increase","user":{"login":"veqryn","id":1897829,"avatar_url":"https://avatars0.githubusercontent.com/u/1897829?v=4","gravatar_id":"","url":"https://api.github.com/users/veqryn","html_url":"https://github.com/veqryn","followers_url":"https://api.github.com/users/veqryn/followers","following_url":"https://api.github.com/users/veqryn/following{/other_user}","gists_url":"https://api.github.com/users/veqryn/gists{/gist_id}","starred_url":"https://api.github.com/users/veqryn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/veqryn/subscriptions","organizations_url":"https://api.github.com/users/veqryn/orgs","repos_url":"https://api.github.com/users/veqryn/repos","events_url":"https://api.github.com/users/veqryn/events{/privacy}","received_events_url":"https://api.github.com/users/veqryn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-02-25T01:00:26Z","updated_at":"2018-02-26T08:51:54Z","closed_at":null,"author_association":"NONE","body":"**What did you do?**\r\nUsed rate / irate / increase to see how much things have increased.  In this case, we had an overall increase in our count by 15.\r\n\r\n**What did you expect to see?**\r\nI expected to see spikes adding up to 15.\r\n\r\n**What did you see instead?**\r\nSpikes adding up to 11.\r\n\r\n**Environment**\r\nServer: Prometheus 2.0.0, Revision 0a74f98628a0463dddc90528220c94de5032d1a0\r\nClient: Prometheus Golang Client library (official)\r\n(all our metrics are initialized and published at their zero-state, as per how the golang client library works)\r\n\r\n**Summary**\r\nI built a bunch of graphs and dashboards, but they just didn't seem to being showing accurate data.  After investigating and comparing with our logs, I confirmed that all of my graphs and dashboards show data that doesn't match what is actually happening in my system.\r\n\r\nThe problem appears to be that when using rate / irate / increase to see the rate that metrics increase, all data/metrics that happen before the 1st scrape are completely ignored/dropped by prometheus.\r\n\r\nWe have a Kubernetes cluster with an auto-scaling cluster.  New pods are constantly being created, and old ones are constantly being killed off, or sometimes getting restarted.\r\nOur scrape interval for everything in our cluster is 60 seconds.\r\n\r\nBecause we auto-scale up and down quite quickly, and because we scale up when we are the busiest, I estimate we lose about 20-40% of our data (not data points, but the actual values; so for example I would be expecting spikes adding up to 5000 over a 12 hour period, yet prometheus shows spikes adding up to 3000, because the largest spikes are all missing).\r\n\r\nFor example, here we show a bunch of 4 new pods starting up, and processing 15 emails:\r\n![image](https://user-images.githubusercontent.com/1897829/36636799-c0e81996-198a-11e8-827c-610b0e910dcc.png)\r\n\r\nBut the spikes add up to only 11, because we are missing the first 4 data points:\r\n![image](https://user-images.githubusercontent.com/1897829/36636808-e4f93766-198a-11e8-9bae-3db94b80d89b.png)\r\n\r\nWhen you don't stack the first graph, you can the problem is because the 4 pods all managed to process 1 email before the first scrape, so the metric came in at 1 for each of them, instead of starting at zero:\r\n![image](https://user-images.githubusercontent.com/1897829/36636821-0c62fa94-198b-11e8-82f0-2ee74a9c03bb.png)\r\n\r\n\r\nI don't think it is reasonable to ask our processes/pods to startup and then sleep for 60 seconds to ensure that a scrape takes place before any work gets done.\r\n\r\nI do not know what the best solution is, but I hope this issue will get addressed, since it makes the rate/irate/increase functions misleading at best, and outright wrong at worst.\r\n\r\nOne potential idea would be to have the client libraries publish a timestamp of when the prometheus library was initialized or when each metric got registered.  Those timestamps would never change for the life of the process/container, so they would only need to be read in and parsed by the prometheus server during the first successful scrape.  The server could then see that the metric actually started at zero, at that timestamp.\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/368430015","html_url":"https://github.com/prometheus/prometheus/issues/3886#issuecomment-368430015","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/3886","id":368430015,"user":{"login":"veqryn","id":1897829,"avatar_url":"https://avatars0.githubusercontent.com/u/1897829?v=4","gravatar_id":"","url":"https://api.github.com/users/veqryn","html_url":"https://github.com/veqryn","followers_url":"https://api.github.com/users/veqryn/followers","following_url":"https://api.github.com/users/veqryn/following{/other_user}","gists_url":"https://api.github.com/users/veqryn/gists{/gist_id}","starred_url":"https://api.github.com/users/veqryn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/veqryn/subscriptions","organizations_url":"https://api.github.com/users/veqryn/orgs","repos_url":"https://api.github.com/users/veqryn/repos","events_url":"https://api.github.com/users/veqryn/events{/privacy}","received_events_url":"https://api.github.com/users/veqryn/received_events","type":"User","site_admin":false},"created_at":"2018-02-26T08:51:54Z","updated_at":"2018-02-26T08:51:54Z","author_association":"NONE","body":"How or why would a counter reset?  My understanding was that a counter resets because the process stopped/died and a new process started.\r\nIn any case, I don't see how a counter reset in this situation would be any different than having a pod that never got scraped at all, which then died.\r\n\r\nLets say you have a process start up, it counts up to 1000, then resets back to zero (or dies and restarts, or w/e), then it counts up to 9000, then resets back to zero again, and then counts up to 5000 and finally gets scraped for the first time.\r\nPrometheus at this point does not know how many times it has been reset (or how many pods out there existed and died before ever getting scraped).\r\nBut it does know that we counted from at least 0 to 5000.\r\nGiven this situation, Prometheus should show that increase to 5000 somehow.  The idea that we would rather show absolutely nothing instead of 5000 does not make any sense to me.\r\n"}}
{"actor":{"display_login":"veqryn"}}