 <p>Try following:</p> <pre><code>import re lines = [ 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', 'Mozilla/5.0 (compatible; AhrefsBot/4.0; +http://ahrefs.com/robot/)', 'msnbot/2.0b (+http://search.msn.com/msnbot.htm)' ] botname = re.compile('\w+bot/[.\w]+', flags=re.IGNORECASE) for line in lines: matched = botname.search(line) if matched: print(matched.group()) </code></pre> <p>prints</p> <pre class="lang-none prettyprint-override"><code>Googlebot/2.1 AhrefsBot/4.0 msnbot/2.0b </code></pre> <p>assumed that bot agent names contain <code>bot/</code>.</p>