{"action":"opened","issue":{"url":"https://api.github.com/repos/flynn/flynn/issues/2787","repository_url":"https://api.github.com/repos/flynn/flynn","labels_url":"https://api.github.com/repos/flynn/flynn/issues/2787/labels{/name}","comments_url":"https://api.github.com/repos/flynn/flynn/issues/2787/comments","events_url":"https://api.github.com/repos/flynn/flynn/issues/2787/events","html_url":"https://github.com/flynn/flynn/issues/2787","id":153333959,"number":2787,"title":"Cluster got wonky for a few minutes on deployment of an application with large repository","user":{"login":"philiplb","id":1672238,"avatar_url":"https://avatars.githubusercontent.com/u/1672238?v=3","gravatar_id":"","url":"https://api.github.com/users/philiplb","html_url":"https://github.com/philiplb","followers_url":"https://api.github.com/users/philiplb/followers","following_url":"https://api.github.com/users/philiplb/following{/other_user}","gists_url":"https://api.github.com/users/philiplb/gists{/gist_id}","starred_url":"https://api.github.com/users/philiplb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philiplb/subscriptions","organizations_url":"https://api.github.com/users/philiplb/orgs","repos_url":"https://api.github.com/users/philiplb/repos","events_url":"https://api.github.com/users/philiplb/events{/privacy}","received_events_url":"https://api.github.com/users/philiplb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-05-05T21:49:06Z","updated_at":"2016-05-05T21:49:06Z","closed_at":null,"body":"Hi,\r\n\r\nthis ticket is maybe more a for your information.\r\n\r\nI tried to deploy an application with a large git history. Large history means a git push of 1.11GB (there used to be some large binary files in the repo some years ago) which should result in a slug of the size ~155MB.\r\n\r\nI pushed the application, the mvn build failed due to some missing dependencies. The last line of the git push is this:\r\n\r\n> remote: Build failed: exec: job exited with status 1\r\n\r\nAnd now the push hangs until I hit Ctrl+C. According to the status endpoint, this systems are unhealthy now in this moment:\r\n- blobstore\r\n- dashboard\r\n- gitreceive\r\n- logaggregator\r\n- postgres\r\n\r\nEverything recovered after a few minutes.\r\n\r\nI tried to delete the application and got an error:\r\n\r\n> timed out waiting for app deletion\r\n\r\nSubsequent deletion tries lead to the same error.\r\n\r\nWithin this time, the other already deployed applications didn't answer or were utterly slow. Note: They don't use the DB resources Flynn offers but are talking to another DB server.\r\n\r\nAfter a few more minutes, the application was gone as expected and everything started to work again.\r\n\r\nYesterday, I even broke the whole cluster by trying to deploy the application:\r\n- Deletion threw a 503 and something else (I can't remember)\r\n- I tried a reboot of a node\r\n- basically all apps stopped to answer\r\n\r\nI restored from backup and sadly don't have logs. Tried to reproduce this, but today, Flynn behaved fine."}}