{"action":"closed","issue":{"url":"https://api.github.com/repos/flynn/flynn/issues/3487","repository_url":"https://api.github.com/repos/flynn/flynn","labels_url":"https://api.github.com/repos/flynn/flynn/issues/3487/labels{/name}","comments_url":"https://api.github.com/repos/flynn/flynn/issues/3487/comments","events_url":"https://api.github.com/repos/flynn/flynn/issues/3487/events","html_url":"https://github.com/flynn/flynn/issues/3487","id":182129422,"number":3487,"title":"High CPU causes cluster failures but no node eviction","user":{"login":"temujin9","id":627414,"avatar_url":"https://avatars.githubusercontent.com/u/627414?v=3","gravatar_id":"","url":"https://api.github.com/users/temujin9","html_url":"https://github.com/temujin9","followers_url":"https://api.github.com/users/temujin9/followers","following_url":"https://api.github.com/users/temujin9/following{/other_user}","gists_url":"https://api.github.com/users/temujin9/gists{/gist_id}","starred_url":"https://api.github.com/users/temujin9/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/temujin9/subscriptions","organizations_url":"https://api.github.com/users/temujin9/orgs","repos_url":"https://api.github.com/users/temujin9/repos","events_url":"https://api.github.com/users/temujin9/events{/privacy}","received_events_url":"https://api.github.com/users/temujin9/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-10T22:22:35Z","updated_at":"2016-11-22T17:54:47Z","closed_at":"2016-11-22T17:54:47Z","body":"The AWS S3 upload step of our backup scripts is getting caught in some loop, spiking CPU through the roof, and rendering a node mostly inoperable. This has caused several outages across multiple clusters.\n\nThis has several unfortunate side effects:\n- The failure is below the threshold of AWS's auto-termination (preventing the issue from autohealing)\n- The high CPU usage renders the node unavailable to SSH for diagnostic purposes\n- The node is still semi-functional, so it is still participating in the overall Flynn cluster.\n\nRemediation is fairly simple, in our environment: kill the stuck node, and wait for it to be replaced. However, Flynn doesn't evict the node even though its involvement is bringing down the other nodes; in one case, a long enough outage resulted in a cluster which could not recover at all, even after the offending node was removed.\n\n@josephglanville worked with me to debug this, and can describe some of how the failures looked from within Flynn.\n"}}
{"actor":{"display_login":"temujin9"}}