{"action":"created","issue":{"url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2644","repository_url":"https://api.github.com/repos/ClusterHQ/flocker","labels_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2644/labels{/name}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2644/comments","events_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2644/events","html_url":"https://github.com/ClusterHQ/flocker/pull/2644","id":134043524,"number":2644,"title":"Benchmark back off when not converged floc 4059","user":{"login":"mpgomez","id":678471,"avatar_url":"https://avatars.githubusercontent.com/u/678471?v=3","gravatar_id":"","url":"https://api.github.com/users/mpgomez","html_url":"https://github.com/mpgomez","followers_url":"https://api.github.com/users/mpgomez/followers","following_url":"https://api.github.com/users/mpgomez/following{/other_user}","gists_url":"https://api.github.com/users/mpgomez/gists{/gist_id}","starred_url":"https://api.github.com/users/mpgomez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpgomez/subscriptions","organizations_url":"https://api.github.com/users/mpgomez/orgs","repos_url":"https://api.github.com/users/mpgomez/repos","events_url":"https://api.github.com/users/mpgomez/events{/privacy}","received_events_url":"https://api.github.com/users/mpgomez/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2016-02-16T17:26:04Z","updated_at":"2016-02-24T08:17:06Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2644","html_url":"https://github.com/ClusterHQ/flocker/pull/2644","diff_url":"https://github.com/ClusterHQ/flocker/pull/2644.diff","patch_url":"https://github.com/ClusterHQ/flocker/pull/2644.patch"},"body":"When waiting for convergence, we were looping really fast (every 0.1 seconds) checking the state.\r\nIt was a problem for bulk creation and deletion of datasets, as we were overloading AWS.\r\n\r\nI have added an exponential backoff. I may make sense to retry fast the first one or two times, but if it is saturated, it is better to loop slower.\r\n\r\nIt improves the issues for bulk creation and deletion (tried with 28 node cluster with 20 containers each, only 3 failures; we were having ~150 failures)"},"comment":{"url":"https://api.github.com/repos/ClusterHQ/flocker/issues/comments/188135001","html_url":"https://github.com/ClusterHQ/flocker/pull/2644#issuecomment-188135001","issue_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2644","id":188135001,"user":{"login":"brendan-donegan","id":2405934,"avatar_url":"https://avatars.githubusercontent.com/u/2405934?v=3","gravatar_id":"","url":"https://api.github.com/users/brendan-donegan","html_url":"https://github.com/brendan-donegan","followers_url":"https://api.github.com/users/brendan-donegan/followers","following_url":"https://api.github.com/users/brendan-donegan/following{/other_user}","gists_url":"https://api.github.com/users/brendan-donegan/gists{/gist_id}","starred_url":"https://api.github.com/users/brendan-donegan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendan-donegan/subscriptions","organizations_url":"https://api.github.com/users/brendan-donegan/orgs","repos_url":"https://api.github.com/users/brendan-donegan/repos","events_url":"https://api.github.com/users/brendan-donegan/events{/privacy}","received_events_url":"https://api.github.com/users/brendan-donegan/received_events","type":"User","site_admin":false},"created_at":"2016-02-24T08:17:06Z","updated_at":"2016-02-24T08:17:06Z","body":"Yes, there is setup-cluster, this is what we were normally using\n\nOn Wed, 24 Feb 2016, 01:45 Marcus Ewert <notifications@github.com> wrote:\n\n> Well, that failed quicker, but I still hit exceptions.OSError: [Errno 24]\n> Too many open files. So I guess I'll bump up my file limits. Although\n> that might wait until tomorrow as I think I'm done for the day.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/ClusterHQ/flocker/pull/2644#issuecomment-188004130>.\n>\n-- \n\nBrendan Donegan - 07771376827\nLead QA Engineer @ ClusterHQ\n"}}