{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/3487","repository_url":"https://api.github.com/repos/prometheus/prometheus","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/3487/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/3487/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/3487/events","html_url":"https://github.com/prometheus/prometheus/issues/3487","id":274710680,"number":3487,"title":"'compaction failed' - prometheus suddenly ate up entire disk","user":{"login":"gregorycerna","id":1924624,"avatar_url":"https://avatars0.githubusercontent.com/u/1924624?v=4","gravatar_id":"","url":"https://api.github.com/users/gregorycerna","html_url":"https://github.com/gregorycerna","followers_url":"https://api.github.com/users/gregorycerna/followers","following_url":"https://api.github.com/users/gregorycerna/following{/other_user}","gists_url":"https://api.github.com/users/gregorycerna/gists{/gist_id}","starred_url":"https://api.github.com/users/gregorycerna/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gregorycerna/subscriptions","organizations_url":"https://api.github.com/users/gregorycerna/orgs","repos_url":"https://api.github.com/users/gregorycerna/repos","events_url":"https://api.github.com/users/gregorycerna/events{/privacy}","received_events_url":"https://api.github.com/users/gregorycerna/received_events","type":"User","site_admin":false},"labels":[{"id":365567268,"url":"https://api.github.com/repos/prometheus/prometheus/labels/component/local%20storage","name":"component/local storage","color":"0052cc","default":false},{"id":365563588,"url":"https://api.github.com/repos/prometheus/prometheus/labels/kind/bug","name":"kind/bug","color":"ff0000","default":false},{"id":365563606,"url":"https://api.github.com/repos/prometheus/prometheus/labels/priority/P1","name":"priority/P1","color":"eb6420","default":false}],"state":"closed","locked":false,"assignee":{"login":"gouthamve","id":7354143,"avatar_url":"https://avatars0.githubusercontent.com/u/7354143?v=4","gravatar_id":"","url":"https://api.github.com/users/gouthamve","html_url":"https://github.com/gouthamve","followers_url":"https://api.github.com/users/gouthamve/followers","following_url":"https://api.github.com/users/gouthamve/following{/other_user}","gists_url":"https://api.github.com/users/gouthamve/gists{/gist_id}","starred_url":"https://api.github.com/users/gouthamve/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gouthamve/subscriptions","organizations_url":"https://api.github.com/users/gouthamve/orgs","repos_url":"https://api.github.com/users/gouthamve/repos","events_url":"https://api.github.com/users/gouthamve/events{/privacy}","received_events_url":"https://api.github.com/users/gouthamve/received_events","type":"User","site_admin":false},"assignees":[{"login":"gouthamve","id":7354143,"avatar_url":"https://avatars0.githubusercontent.com/u/7354143?v=4","gravatar_id":"","url":"https://api.github.com/users/gouthamve","html_url":"https://github.com/gouthamve","followers_url":"https://api.github.com/users/gouthamve/followers","following_url":"https://api.github.com/users/gouthamve/following{/other_user}","gists_url":"https://api.github.com/users/gouthamve/gists{/gist_id}","starred_url":"https://api.github.com/users/gouthamve/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gouthamve/subscriptions","organizations_url":"https://api.github.com/users/gouthamve/orgs","repos_url":"https://api.github.com/users/gouthamve/repos","events_url":"https://api.github.com/users/gouthamve/events{/privacy}","received_events_url":"https://api.github.com/users/gouthamve/received_events","type":"User","site_admin":false}],"milestone":null,"comments":23,"created_at":"2017-11-17T00:21:48Z","updated_at":"2018-05-08T07:01:46Z","closed_at":"2017-11-30T07:28:20Z","author_association":"NONE","body":"<!--\r\n\r\n    Please do *NOT* ask usage questions in Github issues.\r\n\r\n    If your issue is not a feature request or bug report use:\r\n    https://groups.google.com/forum/#!forum/prometheus-users. If\r\n    you are unsure whether you hit a bug, search and ask in the\r\n    mailing list first.\r\n\r\n    You can find more information at: https://prometheus.io/community/\r\n\r\n-->\r\n\r\n**What did you do?**\r\n\r\nFour or five days ago, I upgraded to Prometheus V2, running in a 4-node docker swarm\r\n\r\n**What did you expect to see?**\r\n\r\nPrometheus metrics data to grow fairly slowly, at roughly the same rate as with v1.8 (~1gb/month)\r\n\r\n**What did you see instead? Under which circumstances?**\r\n\r\nIn the past 24 hours, the size of my prometheus data suddenly and inexplicably increased more than 1500x, from ~500mb to 771gb, completely filling up my disk. \r\n \r\n**Environment**\r\n\r\nI'm not sure what caused this, as I haven't modified any of prometheus's configs since I got v2 up and running smoothly. I'm running prometheus in a docker container in swarm mode, so my best guess is that something got corrupted when its container was killed and subsequently restarted on another host. Prometheus's data is being stored on an nfs share available to all hosts, which is then mounted to the container. When checking the data folder, the vast majority of folder in it are `<randomhash>.tmp` folders - the only other files besides the tmp folders are two folders with hashes for names (but no `.tmp`), along with a `wal` folder and a `lock` file.\r\n\r\n\r\n* System information:\r\n\r\n\t`Linux 4.13.0-1-amd64 x86_64`\r\n\r\n* Prometheus version:\r\n\r\n```\r\n/prometheus $ prometheus --version\r\nprometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)\r\n  build user:       root@615b82cb36b6\r\n  build date:       20171108-07:11:59\r\n  go version:       go1.9.2\r\n```\r\n* Docker-compose configuration:\r\n```\r\nversion: '3.3'\r\nservices:\r\n  prom:\r\n    image: prom/prometheus:v2.0.0\r\n    volumes:\r\n      - /docker/prometheus/config:/etc/prometheus\r\n      - /docker/prometheus/data:/prometheus\r\n    networks:\r\n      - monitoring\r\n    ports:\r\n      - 9090:9090\r\n```\r\n\r\n* Prometheus configuration file:\r\n```\r\nglobal:\r\n  scrape_interval:     30s\r\n  evaluation_interval: 30s\r\n\r\n  external_labels:\r\n    monitor: 'prometheus'\r\n\r\nrule_files:\r\n  - \"alert.rules_nodes.yml\"\r\n  - \"alert.rules_tasks.yml\"\r\n  - \"alert.rules_service-groups.yml\"\r\n\r\nalerting:\r\n  alertmanagers:\r\n  - dns_sd_configs:\r\n    - names:\r\n      - 'alerts'\r\n      type: 'A'\r\n      port: 9093\r\n\r\nscrape_configs:\r\n- job_name: 'prometheus'\r\n  dns_sd_configs:\r\n  - names:\r\n    - 'tasks.prom'\r\n    type: 'A'\r\n    port: 9090\r\n\r\n- job_name: 'cadvisor'\r\n  dns_sd_configs:\r\n  - names:\r\n    - 'tasks.cadvisor'\r\n    type: 'A'\r\n    port: 8080\r\n\r\n- job_name: 'node-exporter'\r\n  dns_sd_configs:\r\n  - names:\r\n    - 'tasks.node-exporter'\r\n    type: 'A'\r\n    port: 9100\r\n\r\n- job_name: 'docker-exporter'\r\n  static_configs:\r\n  - targets:\r\n    - 'node1:4999'\r\n    - 'node2:4999'\r\n    - 'node3:4999'\r\n    - 'node4:4999'\r\n\r\n- job_name: 'unifi-exporter'\r\n  dns_sd_configs:\r\n  - names:\r\n    - 'tasks.unifi-exporter'\r\n    type: 'A'\r\n    port: 9130\r\n```\r\n\r\n* Logs:\r\n(all logs retrieved using `docker service logs monitor_prom`)\r\nLogs on prometheus startup \r\n```\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:01:46.384460037Z caller=main.go:215 msg=\"Starting Prometheus\" version=\"(version=2.0.0, branch=HEAD, revision=0a74f98628a0463dddc90528220c94de5032d1a0)\"\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:01:46.384521845Z caller=main.go:216 build_context=\"(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:11:59)\"\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:01:46.384544054Z caller=main.go:217 host_details=\"(Linux 4.13.0-1-amd64 #1 SMP Debian 4.13.4-2 (2017-10-15) x86_64 3434f87590e0 (none))\"\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:01:46.389948893Z caller=web.go:380 component=web msg=\"Start listening for connections\" address=0.0.0.0:9090\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:01:46.390175381Z caller=main.go:314 msg=\"Starting TSDB\"\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:01:46.40905656Z caller=targetmanager.go:71 component=\"target manager\" msg=\"Starting target manager...\"\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=warn ts=2017-11-16T23:02:34.148096492Z caller=head.go:317 component=tsdb msg=\"unknown series references in WAL samples\" count=21956\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:02:34.181636971Z caller=main.go:326 msg=\"TSDB started\"\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:02:34.181753604Z caller=main.go:394 msg=\"Loading configuration file\" filename=/etc/prometheus/prometheus.yml\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:02:34.260471148Z caller=main.go:371 msg=\"Server is ready to receive requests.\"\r\n```\r\nan example of the countless `compaction failed` errors from before disk was filled\r\n```\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:27:04.730665346Z caller=compact.go:361 component=tsdb msg=\"compact blocks\" count=1 mint=1510704000000 maxt=1510711200000\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=error ts=2017-11-16T23:27:05.400205406Z caller=db.go:260 component=tsdb msg=\"compaction failed\" err=\"persist head block: write compaction: add series: out-of-order series added with label set \\\"{__name__=\\\\\\\"go_gc_duration_seconds\\\\\\\",instance=\\\\\\\"10.0.0.244:9090\\\\\\\",job=\\\\\\\"prometheus\\\\\\\",quantile=\\\\\\\"0\\\\\\\"}\\\"\"\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:28:05.436936297Z caller=compact.go:361 component=tsdb msg=\"compact blocks\" count=1 mint=1510704000000 maxt=1510711200000\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=error ts=2017-11-16T23:28:06.103396123Z caller=db.go:260 component=tsdb msg=\"compaction failed\" err=\"persist head block: write compaction: add series: out-of-order series added with label set \\\"{__name__=\\\\\\\"go_gc_duration_seconds\\\\\\\",instance=\\\\\\\"10.0.0.244:9090\\\\\\\",job=\\\\\\\"prometheus\\\\\\\",quantile=\\\\\\\"0\\\\\\\"}\\\"\"\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:29:06.135866736Z caller=compact.go:361 component=tsdb msg=\"compact blocks\" count=1 mint=1510704000000 maxt=1510711200000\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=error ts=2017-11-16T23:29:06.827149013Z caller=db.go:260 component=tsdb msg=\"compaction failed\" err=\"persist head block: write compaction: add series: out-of-order series added with label set \\\"{__name__=\\\\\\\"go_gc_duration_seconds\\\\\\\",instance=\\\\\\\"10.0.0.244:9090\\\\\\\",job=\\\\\\\"prometheus\\\\\\\",quantile=\\\\\\\"0\\\\\\\"}\\\"\"\r\n```\r\n`compaction failed` errors from after disk was filled\r\n```\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:55:18.555189787Z caller=compact.go:361 component=tsdb msg=\"compact blocks\" count=1 mint=1510704000000 maxt=1510711200000\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=error ts=2017-11-16T23:55:18.555336942Z caller=db.go:260 component=tsdb msg=\"compaction failed\" err=\"persist head block: mkdir /prometheus/01BZ3M464VD8YNKY27ZX8HKX2V.tmp: no space left on device\"\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:56:18.567555044Z caller=compact.go:361 component=tsdb msg=\"compact blocks\" count=1 mint=1510704000000 maxt=1510711200000\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=error ts=2017-11-16T23:56:18.567783423Z caller=db.go:260 component=tsdb msg=\"compaction failed\" err=\"persist head block: mkdir /prometheus/01BZ3M60R7BGV0TDYT9G4A3TRK.tmp: no space left on device\"\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=info ts=2017-11-16T23:57:18.580361477Z caller=compact.go:361 component=tsdb msg=\"compact blocks\" count=1 mint=1510704000000 maxt=1510711200000\r\nmonitor_prom.1.3u83u9j3hljv@node3    | level=error ts=2017-11-16T23:57:18.580538384Z caller=db.go:260 component=tsdb msg=\"compaction failed\" err=\"persist head block: mkdir /prometheus/01BZ3M7VBM62PP3B0QAEBADCHK.tmp: no space left on device\"\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/387305040","html_url":"https://github.com/prometheus/prometheus/issues/3487#issuecomment-387305040","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/3487","id":387305040,"user":{"login":"strowi","id":589662,"avatar_url":"https://avatars3.githubusercontent.com/u/589662?v=4","gravatar_id":"","url":"https://api.github.com/users/strowi","html_url":"https://github.com/strowi","followers_url":"https://api.github.com/users/strowi/followers","following_url":"https://api.github.com/users/strowi/following{/other_user}","gists_url":"https://api.github.com/users/strowi/gists{/gist_id}","starred_url":"https://api.github.com/users/strowi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/strowi/subscriptions","organizations_url":"https://api.github.com/users/strowi/orgs","repos_url":"https://api.github.com/users/strowi/repos","events_url":"https://api.github.com/users/strowi/events{/privacy}","received_events_url":"https://api.github.com/users/strowi/received_events","type":"User","site_admin":false},"created_at":"2018-05-08T07:01:46Z","updated_at":"2018-05-08T07:01:46Z","author_association":"NONE","body":"Yes, here too. Moved to 'emptyDir' and everything seems to be working fine for now. If needed i can provide more info.."}}
{"actor":{"display_login":"strowi"}}