{"action":"opened","issue":{"url":"https://api.github.com/repos/concourse/concourse/issues/565","repository_url":"https://api.github.com/repos/concourse/concourse","labels_url":"https://api.github.com/repos/concourse/concourse/issues/565/labels{/name}","comments_url":"https://api.github.com/repos/concourse/concourse/issues/565/comments","events_url":"https://api.github.com/repos/concourse/concourse/issues/565/events","html_url":"https://github.com/concourse/concourse/issues/565","id":168849699,"number":565,"title":"baggageclaim disk usage growing without bounds","user":{"login":"alext","id":5560,"avatar_url":"https://avatars.githubusercontent.com/u/5560?v=3","gravatar_id":"","url":"https://api.github.com/users/alext","html_url":"https://github.com/alext","followers_url":"https://api.github.com/users/alext/followers","following_url":"https://api.github.com/users/alext/following{/other_user}","gists_url":"https://api.github.com/users/alext/gists{/gist_id}","starred_url":"https://api.github.com/users/alext/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alext/subscriptions","organizations_url":"https://api.github.com/users/alext/orgs","repos_url":"https://api.github.com/users/alext/repos","events_url":"https://api.github.com/users/alext/events{/privacy}","received_events_url":"https://api.github.com/users/alext/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-02T10:18:07Z","updated_at":"2016-08-02T10:18:07Z","closed_at":null,"body":"We're running into issues with our concourse deploy running out of space in `/var/vcap/data/baggageclaim/volumes`. It's an 85Gig filesystem and it's full. When this happens we're seeing `volume mounted to container is missing` errors in the UI, and in the atc logs.\r\n\r\nWhen the disk fills up, the pipeline errors all over the place. a few minutes after this, the disk space will drop drastically, but the `volume mounted to container is missing` error is still reported. Eventually (around 30mins later) it seems to recover, and will work again, but the disk usage starts to grow again until it repeats this.\r\n\r\n![disk_usage](https://cloud.githubusercontent.com/assets/5560/17325296/bba713e2-58a2-11e6-975a-9c455c6384f4.png)\r\n\r\nWe're running version: 1.5.1 with https://github.com/concourse/concourse/pull/561 applied. It's deployed by bosh-init using [this manifest](https://github.com/alphagov/paas-cf/blob/prod-0.0.171/manifests/concourse-manifest/concourse-base.yml).\r\n\r\nWhen it gets full we're seeing lots of duplicate volumes in the output of `fly volumes`. The full output can be seen [here]( https://gist.github.com/alext/0b95b50e5d93f401276763315052bc25) (Note: because we deploy with bosh-init, the worker is named \"unknown\").\r\n\r\nWe have the following pipelines: https://github.com/alphagov/paas-cf/blob/prod-0.0.171/concourse/pipelines/create-bosh-cloudfoundry.yml and https://github.com/alphagov/paas-cf/blob/prod-0.0.171/concourse/pipelines/failure-testing.yml (the failure-testing one isn't run regularly)\r\n\r\nI first raised this on slack (here: https://concourseci.slack.com/archives/general/p1470062924003131) where it was suggested that I raise this issue."}}
{"actor":{"display_login":"alext"}}