{"action":"created","issue":{"url":"https://api.github.com/repos/wal-e/wal-e/issues/283","repository_url":"https://api.github.com/repos/wal-e/wal-e","labels_url":"https://api.github.com/repos/wal-e/wal-e/issues/283/labels{/name}","comments_url":"https://api.github.com/repos/wal-e/wal-e/issues/283/comments","events_url":"https://api.github.com/repos/wal-e/wal-e/issues/283/events","html_url":"https://github.com/wal-e/wal-e/issues/283","id":183217586,"number":283,"title":"Confused by disk space usage","user":{"login":"boncey","id":265604,"avatar_url":"https://avatars.githubusercontent.com/u/265604?v=3","gravatar_id":"","url":"https://api.github.com/users/boncey","html_url":"https://github.com/boncey","followers_url":"https://api.github.com/users/boncey/followers","following_url":"https://api.github.com/users/boncey/following{/other_user}","gists_url":"https://api.github.com/users/boncey/gists{/gist_id}","starred_url":"https://api.github.com/users/boncey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/boncey/subscriptions","organizations_url":"https://api.github.com/users/boncey/orgs","repos_url":"https://api.github.com/users/boncey/repos","events_url":"https://api.github.com/users/boncey/events{/privacy}","received_events_url":"https://api.github.com/users/boncey/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-10-15T16:12:19Z","updated_at":"2016-10-15T21:03:42Z","closed_at":null,"body":"I'm confused by the amount of disk space that my wal-e backups are taking up (and the associated AWS bill).\r\n\r\nThe total size of all my databases is a smidgen over 3GB.\r\nYet after running `wal-e` on an empty bucket for 12 days the total disk usage of that folder is`161GB`.\r\n\r\nI run a full `backup-push` every week.\r\nAlso, I run a weekly job to delete all but the last five backups.\r\n\r\n    /usr/local/bin/wal-e --terse delete --confirm retain 5\r\n\r\nA few weeks back my S3 bucket had grown to half a terrabyte.\r\nI ran the command to delete everything but after it had completed I still had a folder with `345GB` of data.\r\nI cleaned that up by hand to start again from scratch.\r\n\r\nWhen I now run the command to show backup size I see that I have three backups each around `4GB` in size.\r\nYet the total usage is `161GB` and rising.\r\n\r\nHere are the relevant settings from my `postgresql.conf`.\r\n```\r\nwal_buffers = 16MB\r\nwal_level = archive\r\narchive_mode = on\r\narchive_command = 'wal-e --terse wal-push %p'\r\narchive_timeout = 60\r\n```\r\n\r\nIs all this normal (and if so, why) or am I doing something wrong?\r\nThanks, Darren."},"comment":{"url":"https://api.github.com/repos/wal-e/wal-e/issues/comments/254010955","html_url":"https://github.com/wal-e/wal-e/issues/283#issuecomment-254010955","issue_url":"https://api.github.com/repos/wal-e/wal-e/issues/283","id":254010955,"user":{"login":"deverant","id":180004,"avatar_url":"https://avatars.githubusercontent.com/u/180004?v=3","gravatar_id":"","url":"https://api.github.com/users/deverant","html_url":"https://github.com/deverant","followers_url":"https://api.github.com/users/deverant/followers","following_url":"https://api.github.com/users/deverant/following{/other_user}","gists_url":"https://api.github.com/users/deverant/gists{/gist_id}","starred_url":"https://api.github.com/users/deverant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deverant/subscriptions","organizations_url":"https://api.github.com/users/deverant/orgs","repos_url":"https://api.github.com/users/deverant/repos","events_url":"https://api.github.com/users/deverant/events{/privacy}","received_events_url":"https://api.github.com/users/deverant/received_events","type":"User","site_admin":false},"created_at":"2016-10-15T21:03:42Z","updated_at":"2016-10-15T21:03:42Z","body":"Apart from the backups you do with `backup-push` you are also uploading every WAL (Write-ahead log) file to S3 (that is what the archive_command does). These files contain basically every modification to your database and allow doing point-in-time-recovery of the database.\r\n\r\nSo my first guess would be that you have a lot of write traffic in the database so you end up uploading a lot of WAL files into S3. What `delete` will do for you is delete WAL files older than any of the full backups left in storage."}}
{"actor":{"display_login":"deverant"}}