{"action":"created","issue":{"url":"https://api.github.com/repos/flynn/flynn/issues/3163","repository_url":"https://api.github.com/repos/flynn/flynn","labels_url":"https://api.github.com/repos/flynn/flynn/issues/3163/labels{/name}","comments_url":"https://api.github.com/repos/flynn/flynn/issues/3163/comments","events_url":"https://api.github.com/repos/flynn/flynn/issues/3163/events","html_url":"https://github.com/flynn/flynn/issues/3163","id":166810524,"number":3163,"title":"Default AWS cluster disk full (after almost no usage)","user":{"login":"IsNull","id":439313,"avatar_url":"https://avatars.githubusercontent.com/u/439313?v=3","gravatar_id":"","url":"https://api.github.com/users/IsNull","html_url":"https://github.com/IsNull","followers_url":"https://api.github.com/users/IsNull/followers","following_url":"https://api.github.com/users/IsNull/following{/other_user}","gists_url":"https://api.github.com/users/IsNull/gists{/gist_id}","starred_url":"https://api.github.com/users/IsNull/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IsNull/subscriptions","organizations_url":"https://api.github.com/users/IsNull/orgs","repos_url":"https://api.github.com/users/IsNull/repos","events_url":"https://api.github.com/users/IsNull/events{/privacy}","received_events_url":"https://api.github.com/users/IsNull/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-07-21T12:48:06Z","updated_at":"2016-07-21T14:35:49Z","closed_at":null,"body":"I just used a AWS single node cluster (with the default 2x 50GB disks, `v20160624.1`) for some testing with around 5 applications, and now the disk appears to be full:\r\n\r\n```\r\nCloning into '/tmp/app'...\r\n-----> Building shipping-service...\r\n-----> Gradle app detected\r\nmkdir: cannot create directory '/tmp/jvm-common': No space left on device\r\nERROR: Build failed: exec: job exited with status 1\r\nexit status 1\r\n```\r\n\r\nThe apps result in slugs usually around 150MB (Java stuff)\r\n\r\nOn this host I see:\r\n\r\n```\r\n  System information as of Tue Jul 12 17:48:58 UTC 2016\r\n\r\n  System load:  0.13               Users logged in:         0\r\n  Usage of /:   46.2% of 49.08GB   IP address for eth0:     10.0.4.8\r\n  Memory usage: 81%                IP address for docker0:  172.17.0.1\r\n  Swap usage:   0%                 IP address for flynnbr0: 100.100.12.1\r\n  Processes:    610\r\n```\r\nand\r\n```\r\nubuntu@ip-10-x-x-x:~$ df -Th\r\nFilesystem                                         Type      Size  Used Avail Use% Mounted on\r\nudev                                               devtmpfs  2.0G   12K  2.0G   1% /dev\r\ntmpfs                                              tmpfs     396M  560K  395M   1% /run\r\n/dev/xvda1                                         ext4       50G   48G     0 100% /\r\nnone                                               tmpfs     4.0K     0  4.0K   0% /sys/fs/cgroup\r\nnone                                               tmpfs     5.0M     0  5.0M   0% /run/lock\r\nnone                                               tmpfs     2.0G   74M  1.9G   4% /run/shm\r\nnone                                               tmpfs     100M     0  100M   0% /run/user\r\nflynn-default                                      zfs        27G     0   27G   0% /flynn-default\r\nflynn-default/4a8d2f49-2470-4fec-9f07-a855c0da282e zfs        27G  128K   27G   1% /var/lib/flynn/volumes/zfs/mnt/4a8d2f49-2470-4fec-9f07-a855c0da282e\r\nflynn-default/fcd43a04-2572-45a3-b946-44273a964eee zfs        49G   22G   27G  45% /var/lib/flynn/volumes/zfs/mnt/fcd43a04-2572-45a3-b946-44273a964eee\r\n```\r\n\r\nThe debug log seems to be quite big as well:\r\n\r\n```\r\nINFO[07-21|12:42:17] uploading logs and debug information to a private, anonymous gist \r\nINFO[07-21|12:42:17] this may take a while depending on the size of your logs \r\nINFO[07-21|12:42:17] getting flynn-host logs \r\nINFO[07-21|12:42:17] getting job logs \r\nINFO[07-21|12:42:46] getting system information \r\nINFO[07-21|12:42:46] Total size of 257420877 bytes exceeds gist limit, falling back to tarball. \r\n```\r\n\r\nWhat could cause this?  If you really need the 25MB tarball I can upload it somewhere.\r\n\r\n**Edit:** Uploaded the debug logs\r\n[debug.tar.gz](https://github.com/flynn/flynn/files/376107/debug.tar.gz)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/flynn/flynn/issues/comments/234273239","html_url":"https://github.com/flynn/flynn/issues/3163#issuecomment-234273239","issue_url":"https://api.github.com/repos/flynn/flynn/issues/3163","id":234273239,"user":{"login":"IsNull","id":439313,"avatar_url":"https://avatars.githubusercontent.com/u/439313?v=3","gravatar_id":"","url":"https://api.github.com/users/IsNull","html_url":"https://github.com/IsNull","followers_url":"https://api.github.com/users/IsNull/followers","following_url":"https://api.github.com/users/IsNull/following{/other_user}","gists_url":"https://api.github.com/users/IsNull/gists{/gist_id}","starred_url":"https://api.github.com/users/IsNull/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IsNull/subscriptions","organizations_url":"https://api.github.com/users/IsNull/orgs","repos_url":"https://api.github.com/users/IsNull/repos","events_url":"https://api.github.com/users/IsNull/events{/privacy}","received_events_url":"https://api.github.com/users/IsNull/received_events","type":"User","site_admin":false},"created_at":"2016-07-21T14:35:48Z","updated_at":"2016-07-21T14:35:48Z","body":"Ok I got the script to run without creating a actual file:\r\n`bash <(curl -s https://gist.githubusercontent.com/lmars/6d072bc88c3a4c9f87d2/raw/49f0efbfcc5433fea9fded5e5c0baf39d1ad9c06/flynn-cleanup.sh)`\r\n\r\nThis gives me output like\r\n\r\n```\r\n[...]\r\nrm: cannot remove '/var/lib/docker/aufs/mnt/tmp-ip10048-0d0f065c-65ba-474a-a706-27aadbf92095': Device or resource busy\r\nrm: cannot remove '/var/lib/docker/aufs/mnt/tmp-ip10048-92c88cbd-5dcd-491f-a003-41fd90b590ec': Device or resource busy\r\n```\r\n\r\nStill running ..."}}
{"actor":{"display_login":"IsNull"}}