{"action":"created","issue":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/7712","repository_url":"https://api.github.com/repos/influxdata/influxdb","labels_url":"https://api.github.com/repos/influxdata/influxdb/issues/7712/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/influxdb/issues/7712/comments","events_url":"https://api.github.com/repos/influxdata/influxdb/issues/7712/events","html_url":"https://github.com/influxdata/influxdb/issues/7712","id":194807104,"number":7712,"title":"[BUG] Starting Influx eats up all available disk space, error with compaction?","user":{"login":"mebezac","id":4645899,"avatar_url":"https://avatars.githubusercontent.com/u/4645899?v=3","gravatar_id":"","url":"https://api.github.com/users/mebezac","html_url":"https://github.com/mebezac","followers_url":"https://api.github.com/users/mebezac/followers","following_url":"https://api.github.com/users/mebezac/following{/other_user}","gists_url":"https://api.github.com/users/mebezac/gists{/gist_id}","starred_url":"https://api.github.com/users/mebezac/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mebezac/subscriptions","organizations_url":"https://api.github.com/users/mebezac/orgs","repos_url":"https://api.github.com/users/mebezac/repos","events_url":"https://api.github.com/users/mebezac/events{/privacy}","received_events_url":"https://api.github.com/users/mebezac/received_events","type":"User","site_admin":false},"labels":[{"id":491812597,"url":"https://api.github.com/repos/influxdata/influxdb/labels/area/storage","name":"area/storage","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-12-11T02:02:35Z","updated_at":"2016-12-15T18:03:18Z","closed_at":null,"body":"Suddenly and unexpectedly, Influx started using up all disk space and stopping Influx frees up that disk space, while starting Influx takes up all available space again.\r\n\r\nWhen Influx is stopped: \r\n![image](https://cloud.githubusercontent.com/assets/4645899/21077417/d3cc935c-bf18-11e6-9de2-67943738b339.png)\r\n\r\nWhen Influx is started and run for a couple minutes:\r\n![image](https://cloud.githubusercontent.com/assets/4645899/21077419/e4af5542-bf18-11e6-9115-c1677b1dc51a.png)\r\n\r\nHere is a graph of the disk usage.\r\n![image](https://cloud.githubusercontent.com/assets/4645899/21077411/8a5579b4-bf18-11e6-8693-c6e688cfa59e.png)\r\n\r\nYou can see the disk usage keeps jumping to 100% (it only falls when I restart Influx)\r\n\r\nOnce the disk space is full the log starts being filled with lines like: \r\n\r\n```\r\n[tsm1] 2016/12/11 01:27:40 beginning level 2 compaction of group 0, 1 TSM files\r\n[tsm1] 2016/12/11 01:27:40 compacting level 2 group (0) /vol/influxdb/data/telegraf/default/160/000000515-000000002.tsm (#0)\r\n[tsm1] 2016/12/11 01:27:40 aborted full compaction group (0). compaction in progress\r\n[tsm1] 2016/12/11 01:27:40 aborted level 2 compaction group (0). compaction in progress\r\n[tsm1] 2016/12/11 01:27:40 beginning level 2 compaction of group 0, 1 TSM files\r\n[tsm1] 2016/12/11 01:27:40 compacting level 2 group (0) /vol/influxdb/data/telegraf/default/144/000000527-000000002.tsm (#0)\r\n[tsm1] 2016/12/11 01:27:40 aborted level 2 compaction group (0). compaction in progress\r\n[tsm1] 2016/12/11 01:27:40 beginning level 2 compaction of group 0, 1 TSM files\r\n[tsm1] 2016/12/11 01:27:40 compacting level 2 group (0) /vol/influxdb/data/telegraf/default/72/000000503-000000002.tsm (#0)\r\n[write] 2016/12/11 01:27:41 write failed for shard 216: engine: error writing WAL entry: write /vol/influxdb/wal/telegraf/default/216/_01550.wal: no space left on device\r\n```\r\n\r\nI also attempted setting `cache-max-memory-size = 4194304000` as I saw an error earlier regarding that, but that did not stop the problem.\r\n\r\n__System info:__ [Include InfluxDB version, operating system name, and other relevant details]\r\nInfluxDB version: 1.1.0\r\noperating system name:  64-bit Ubuntu 14.04.5 LTS\r\nDisk Space for influx data: 50gb\r\n\r\n__Steps to reproduce:__\r\n\r\n1. Unknown, it happened randomly and unexpectedly. \r\n\r\n__Expected behavior:__ For Influx to not take up all available disk space\r\n\r\n__Actual behavior:__ Influx is taking up all available disk space\r\n\r\nThese files available at this [gist](https://gist.github.com/mebezac/571c0b53e785ebcc130dcabb8070efab)\r\n\r\n```\r\ncurl -o block.txt \"http://localhost:8086/debug/pprof/block?debug=1\" \r\ncurl -o goroutine.txt \"http://localhost:8086/debug/pprof/goroutine?debug=1\" \r\ncurl -o heap.txt \"http://localhost:8086/debug/pprof/heap?debug=1\" \r\ncurl -o vars.txt \"http://localhost:8086/debug/vars\" \r\niostat -xd 1 30 > iostat.txt\r\ninflux -execute \"show shards\" > shards.txt\r\ninflux -execute \"show stats\" > stats.txt\r\ninflux -execute \"show diagnostics\" > diagnostics.txt\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/comments/267397495","html_url":"https://github.com/influxdata/influxdb/issues/7712#issuecomment-267397495","issue_url":"https://api.github.com/repos/influxdata/influxdb/issues/7712","id":267397495,"user":{"login":"jwilder","id":219935,"avatar_url":"https://avatars.githubusercontent.com/u/219935?v=3","gravatar_id":"","url":"https://api.github.com/users/jwilder","html_url":"https://github.com/jwilder","followers_url":"https://api.github.com/users/jwilder/followers","following_url":"https://api.github.com/users/jwilder/following{/other_user}","gists_url":"https://api.github.com/users/jwilder/gists{/gist_id}","starred_url":"https://api.github.com/users/jwilder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwilder/subscriptions","organizations_url":"https://api.github.com/users/jwilder/orgs","repos_url":"https://api.github.com/users/jwilder/repos","events_url":"https://api.github.com/users/jwilder/events{/privacy}","received_events_url":"https://api.github.com/users/jwilder/received_events","type":"User","site_admin":false},"created_at":"2016-12-15T18:03:17Z","updated_at":"2016-12-15T18:03:17Z","body":"Your server looks like it might be somewhat underpowered for your workload.  It looks like it only has 2 cpu cores.  Are your disks SSDs?\r\n\r\nThe large number to `*-000000001.tmp` files looks a little odd.  There may be a bug where failed snapshots are not cleaning up the `tmp` files for some reason.  Do you have any errors in the logs related to snapshotting?  A large number of `tmp` files can happen normally when a large compaction runs, but the naming of the files would be different."}}
{"actor":{"display_login":"jwilder"}}