title: logstash : exclude bots from result
body: <p>I use logstash to store my web server logs into an elasticsearch engine. Into my logstash configuration file I also use the "useragent" plugin to get pretty user agent informations. So a record into ES like this :</p>
  
  <pre><code>    "message": "157.55.XXX.XXX - - [10/Oct/2016:02:24:27 +0200] "GET /handle/boreal:5621?site_name=BOREAL HTTP/1.1" 301 373 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"",
      ...
      "agent": ""Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)""
      "useragent": {
          "name": "bingbot",
          "os": "Other",
          "os_name": "Other",
          "device": "Spider",
          "major": "2",
          "minor": "0"
      } 
  </code></pre>
  
  <p>As you notices, this request is from the Microsoft BingBot robot. I checked all my record and I find a lot of bot accessing my websites : bingbot, googlebot, BaiduSpider, Yahoo! Slurp, ...</p>
  
  <p>I'm now tring to exclude these requests from my ES response. But I didn't find an elegant solution. As I'm very beginner with ES query, can you help me to improve my request ?</p>
  
  <pre><code>{                                               
    "size": 0,                                    
    "query": {                                    
      "filtered": {                               
        "query": {                                
          "match_all": {}                         
        },                                        
        "filter": {                               
          "bool": {                               
            "must_not": [                         
              {                                   
                "regexp": {                       
                  "useragent.name": ".*bot.*"     
                }                                 
              }                                   
            ]                                     
          }                                       
        }                                         
      }                                           
    },                                            
    "aggs": {                                     
      "agent": {                                  
        "terms": {                                
          "field": "useragent.name.raw"           
        }                                         
      }                                           
    }                                             
  }                                               
  </code></pre>
  
  <p>Using this request, all request from bingbot, googlebot, or any other "stuffbot" are exclude but not "Yahoo! Slurp","BaiduSpider", ...
  I tried with more complex regexp "(.*bot.*|BaiduSpider|Yahoo! Slurp)" but Baidu and yahoo results are still present into ES response.</p>