{"action":"created","issue":{"url":"https://api.github.com/repos/fail2ban/fail2ban/issues/1588","repository_url":"https://api.github.com/repos/fail2ban/fail2ban","labels_url":"https://api.github.com/repos/fail2ban/fail2ban/issues/1588/labels{/name}","comments_url":"https://api.github.com/repos/fail2ban/fail2ban/issues/1588/comments","events_url":"https://api.github.com/repos/fail2ban/fail2ban/issues/1588/events","html_url":"https://github.com/fail2ban/fail2ban/issues/1588","id":184697132,"number":1588,"title":"fail2ban slow shutdown + simple fix","user":{"login":"davidfavor","id":87222,"avatar_url":"https://avatars.githubusercontent.com/u/87222?v=3","gravatar_id":"","url":"https://api.github.com/users/davidfavor","html_url":"https://github.com/davidfavor","followers_url":"https://api.github.com/users/davidfavor/followers","following_url":"https://api.github.com/users/davidfavor/following{/other_user}","gists_url":"https://api.github.com/users/davidfavor/gists{/gist_id}","starred_url":"https://api.github.com/users/davidfavor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidfavor/subscriptions","organizations_url":"https://api.github.com/users/davidfavor/orgs","repos_url":"https://api.github.com/users/davidfavor/repos","events_url":"https://api.github.com/users/davidfavor/events{/privacy}","received_events_url":"https://api.github.com/users/davidfavor/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-10-23T15:40:45Z","updated_at":"2016-11-01T21:16:46Z","closed_at":null,"body":"On high traffic sites (1000s/minute requests) fail2ban chokes during shutdown.\n\nBlocking Bots for high traffic sites can generate 1000s of iptables entries which can be deleted individually in a timely fashion, which creates several problems.\n\n1) service fail2ban stop/restart - can require minutes to complete.\n\n2) service fail2ban stop/restart - fails due to timeout expiration on command\n\n3) host reboots - hang for many minutes while many containers are hung in fail2ban, as it attempts to delete each chain sequentially\n# 3 is a real killer, because other services shutdown quickly, so connectivity to Websites running in each container is lost, while fail2ban waits to delete each iptables rule individually.\n\nFix is simple.\n\nFor all fail2ban chains (beginning with \"f2b-\") flush the chain, then delete the chain.\n\nThis logic change means fail2ban completes its entire teardown/shutdown sequence in seconds rather than minutes.\n\nThe simple perl script I use for fail2ban teardown usually executes <5 seconds, where normal fail2ban teardown can take 5-10+ minutes.\n\nImagine 64 containers on a machine, where each fail2ban took 5 minutes to shutdown. Since shutdown time is CPU bound running iptables, each CPU tends to peg toward 100%. If there are 16 threads on the machine, then roughly 100% CPU will each up around...\n\n20 minutes == 5 minutes/fail2ban \\* 4 (64 fail2ban instances @ 100% CPU / 16 threads).\n\nI've begun wiring my script into systemd, so fail2ban-fast-stop (script) runs before normal fail2ban shutdown.\n\nHappy to provide this script to anyone who likes. I'll place my fail2ban-fast-stop script somewhere + add update this ticket shortly.\n"},"comment":{"url":"https://api.github.com/repos/fail2ban/fail2ban/issues/comments/257700395","html_url":"https://github.com/fail2ban/fail2ban/issues/1588#issuecomment-257700395","issue_url":"https://api.github.com/repos/fail2ban/fail2ban/issues/1588","id":257700395,"user":{"login":"AlexisWilke","id":1028617,"avatar_url":"https://avatars.githubusercontent.com/u/1028617?v=3","gravatar_id":"","url":"https://api.github.com/users/AlexisWilke","html_url":"https://github.com/AlexisWilke","followers_url":"https://api.github.com/users/AlexisWilke/followers","following_url":"https://api.github.com/users/AlexisWilke/following{/other_user}","gists_url":"https://api.github.com/users/AlexisWilke/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexisWilke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexisWilke/subscriptions","organizations_url":"https://api.github.com/users/AlexisWilke/orgs","repos_url":"https://api.github.com/users/AlexisWilke/repos","events_url":"https://api.github.com/users/AlexisWilke/events{/privacy}","received_events_url":"https://api.github.com/users/AlexisWilke/received_events","type":"User","site_admin":false},"created_at":"2016-11-01T21:16:46Z","updated_at":"2016-11-01T21:16:46Z","body":"> like `stop_without_unban`\r\n\r\nNow I like that idea. I do not have many IPs and it still takes my test VMs a good 30 seconds to shutdown (although it may be better with 0.10, as you mentioned).\r\n\r\nIt does not seem useful to unban (unless you are testing.) An unban should only occur when the ban for a host times out in the database. I fail to see why it would be required otherwise."}}
{"actor":{"display_login":"AlexisWilke"}}