title: Rate Limit Bot Traffic Using Fail2ban
body: <p>We have configured Fail2ban in Suse-Linux-Enterprise Server for Rate Limiting Bots Traffic.  Below is the configuration done in jail.local file.</p>
  <pre><code>[apache-badbots]
  enabled  = true
  port     = http,https
  filter   = apache-badbots
  action   = iptables-allports[name=apache-badbots, port=&quot;http,https&quot; protocol=tcp]
  logpath  = /var/persistent/apache2/logs/site1-access.log
  findtime  = 60
  bantime   = 600
  maxretry = 1
  </code></pre>
  <p>Below is the Regex Configuration.</p>
  <pre><code>failregex = &lt;HOST&gt; -.*(EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|Googlebot/2\.1)
  </code></pre>
  <p>Below is the Log Format:</p>
  <pre><code>[14/Jul/2020:11:38:09 +0000] 192.168.1.14 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 &quot;GET /sessionValueLink.action?crud=s&amp;keyValue=JsMethodName&amp;insertValue=submitShippingAddress();&amp;dt=Tue%20Jul%2014%202020%2017:08:09%20GMT+0530%20(India%20Standard%20Time) HTTP/1.1&quot; 200 44 [0/1894] &quot;Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/192.168.2.1 Safari/537.36&quot;
  </code></pre>
  <p>When I ran the Regex Test, we are getting the below result.</p>
  <pre><code>fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-badbots.conf
  </code></pre>
  <p><strong>Results</strong></p>
  <pre><code>Failregex: 2438 total
  |-  #) [# of hits] regular expression
  |   1) [2438] &lt;HOST&gt; -.*(Googlebot/2\.1)
  `-
  Ignoreregex: 0 total
  Date template hits:
  |- [# of hits] date format
  |  [113634] Day/MONTH/Year:Hour:Minute:Second
  `-
  Lines: 113634 lines, 0 ignored, 2438 matched, 111196 missed
  Missed line(s): too many to print.  Use --print-all-missed to print all 111196 lines
  </code></pre>
  <p>Issue is when we hit Google Bot continuously we are getting the IP Address is blocked in Log but it is not showing the Valid IPAddress and it is not blocking the bots traffic.  Please find the below Log for your reference.</p>
  <pre><code>2020-07-14 14:17:18,330 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
  2020-07-14 14:17:18,330 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
  2020-07-14 14:17:18,612 fail2ban.actions[431]: INFO    [apache-badbots] 0.0.1.147 already banned
  2020-07-14 14:27:03,274 fail2ban.actions[431]: WARNING [apache-badbots] Unban 0.0.1.147
  2020-07-14 14:38:40,817 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 302 = ['0.0.1.46']
  2020-07-14 14:38:41,073 fail2ban.actions[431]: WARNING [apache-badbots] Ban 0.0.1.46
  2020-07-14 14:39:49,903 fail2ban.filter [431]: WARNING Determined IP using DNS Lookup: 403 = ['0.0.1.147']
  2020-07-14 14:39:50,162 fail2ban.actions[431]: WARNING [apache-badbots] Ban 0.0.1.147
  </code></pre>
  <p>What mistake we have done here? How to solve the issue. I am new to <code>fail2ban</code>, any help will be appreciated.</p>