{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/tsdb/issues/480","repository_url":"https://api.github.com/repos/prometheus/tsdb","labels_url":"https://api.github.com/repos/prometheus/tsdb/issues/480/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/tsdb/issues/480/comments","events_url":"https://api.github.com/repos/prometheus/tsdb/issues/480/events","html_url":"https://github.com/prometheus/tsdb/issues/480","id":391140254,"node_id":"MDU6SXNzdWUzOTExNDAyNTQ=","number":480,"title":"Memory issues on short block sizes","user":{"login":"radeklesniewski","id":15095729,"node_id":"MDQ6VXNlcjE1MDk1NzI5","avatar_url":"https://avatars2.githubusercontent.com/u/15095729?v=4","gravatar_id":"","url":"https://api.github.com/users/radeklesniewski","html_url":"https://github.com/radeklesniewski","followers_url":"https://api.github.com/users/radeklesniewski/followers","following_url":"https://api.github.com/users/radeklesniewski/following{/other_user}","gists_url":"https://api.github.com/users/radeklesniewski/gists{/gist_id}","starred_url":"https://api.github.com/users/radeklesniewski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/radeklesniewski/subscriptions","organizations_url":"https://api.github.com/users/radeklesniewski/orgs","repos_url":"https://api.github.com/users/radeklesniewski/repos","events_url":"https://api.github.com/users/radeklesniewski/events{/privacy}","received_events_url":"https://api.github.com/users/radeklesniewski/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-12-14T14:49:29Z","updated_at":"2018-12-17T21:02:51Z","closed_at":null,"author_association":"NONE","body":"##\r\n**Bug Report**\r\n\r\nWe are testing Prometheus with Thanos to gather metrics from multiple Openshift clusters. On each we have significant amount of ephemeral pods, which are being scraped by Prometheus. We encounter strange memory consumption pattern during test that resembles our real time metrics almost 1:1. We are able to process millions of metrics when changes for particular label occurs every ~2 hours, while for the changes which are happening more often we see constant increase in memory consumption. We tried to lower retention time to 15 minutes, and block size down to 5 minutes, but this didn't help as a lot as we still were seeing the increase in memory consumption, even after the retention period. Further investigation has shown that short block sizes together with short retention caused degradation of performance of writing the blocks to disk. After 30 minutes of test the 5 minutes block was being flushed longer than 10 minutes, +/- 3m for head GC. \r\n\r\nAdditionally, we are observing unexpected behaviour of retention policy execution. As far as we know after debugging Prometheus, there are two conditions for block to be deleted:\r\n1. Block is beyond retention,\r\n2. Block is not opened while dropping old blocks from memory occurs.\r\n\r\nExample of unexpected behaviour: \r\n\tConfig: retention=15m, min-block-duration=5m, max-block-duration=5m\r\n\tBehaviour: old blocks are deleted not after ~20 minutes (retention + newest block duration), but after ~60 minutes (Prometheus instance is not queried while the test is performed, so probably there is no reason for block to be loaded in memory). Test is executed up to ~2 hours and during that time we are observing constant increase in memory consumption, even though after 1st hour of test number of blocks and data series is constant.\r\n\r\nAfter locking label to change every 2 hours, we could easily use longer blocks and retention periods as memory consumption was stable. Memory consumption peaks at 16GB with average at 11GB for 3M metrics per minute, while for labels which were changing every minute for 1.5M after ~30m it crashed at ~100GB.\r\n\r\nIn both cases proportions between number of different metrics to number of blocks was constant, which raises a question, why is it behaving like that and how can we tweak the Prometheus to be able to perform better with high cardinality labels (e.g. cron job id / pod name). We are able to access and aggregate over such metrics in Thanos so Prometheus would need to be able to scrape and preserve data only for a time needed to flush block to disk.\r\n### \r\n**What did you expect to see?**\r\nWe expected to see proportional relation between block sizes, number of different metrics (cardinality / data points), retention and memory consumption.\r\n### \r\n**What did you see instead? Under which circumstances?**\r\nSignificant memory consumption increase for metrics with high cardinality / variable label\r\n### \r\n**Environment**\r\nOpenshift v.3.9 with live data\r\nRHEL 6 VM with artificial data (Linux 2.6.32-696.18.7.el6.x86_64 x86_64)\r\n\r\n    \r\n\r\n- Prometheus version:\r\n    Tested for versions:\r\n    2.3, 2.4, 2.5\r\n\r\n- Prometheus configuration file:\r\n    3000 jobs, 1 minute scrape interval, 1-3M metrics(total)\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/tsdb/issues/comments/447997630","html_url":"https://github.com/prometheus/tsdb/issues/480#issuecomment-447997630","issue_url":"https://api.github.com/repos/prometheus/tsdb/issues/480","id":447997630,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0Nzk5NzYzMA==","user":{"login":"krasi-georgiev","id":8903888,"node_id":"MDQ6VXNlcjg5MDM4ODg=","avatar_url":"https://avatars0.githubusercontent.com/u/8903888?v=4","gravatar_id":"","url":"https://api.github.com/users/krasi-georgiev","html_url":"https://github.com/krasi-georgiev","followers_url":"https://api.github.com/users/krasi-georgiev/followers","following_url":"https://api.github.com/users/krasi-georgiev/following{/other_user}","gists_url":"https://api.github.com/users/krasi-georgiev/gists{/gist_id}","starred_url":"https://api.github.com/users/krasi-georgiev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krasi-georgiev/subscriptions","organizations_url":"https://api.github.com/users/krasi-georgiev/orgs","repos_url":"https://api.github.com/users/krasi-georgiev/repos","events_url":"https://api.github.com/users/krasi-georgiev/events{/privacy}","received_events_url":"https://api.github.com/users/krasi-georgiev/received_events","type":"User","site_admin":false},"created_at":"2018-12-17T21:02:51Z","updated_at":"2018-12-17T21:02:51Z","author_association":"MEMBER","body":"can you maybe add some more specifics about the setup. How many pods are you trying to monitor and how often do these change completely? \r\n\r\nOne suggestion is maybe drop some labels that are unique for every pod and thus reduce the cardinality."}}
{"actor":{"display_login":"krasi-georgiev"}}