title: Unsuported browser agent when crawling TripAdvisor with boilerpipe
body: <p>I'm programming a generic webcrawler that gets the main content from a given webpage (it has to crawl different pages).</p>
  <p>I've tried to achieve this with different tools, among them:</p>
  <ul>
  <li><strong>HtmlUnit</strong>: returned me too much scrap when crawling.</li>
  <li><strong>Essence</strong>: failed to get the important information on many pages.</li>
  <li><strong>Boilerpipe</strong>: retrieves the content successfully, almost perfect results but:</li>
  </ul>
  <p>When I try to crawl pages like TripAdvisor instead of the given webpage html it returns the following message:</p>
  <blockquote>
  <p>We noticed that you're using an unsupported browser. The Tripadvisor
  website may not display properly.We support the following browsers:
  Windows: Internet Explorer, Mozilla Firefox, Google Chrome. Mac:
  Safari.</p>
  </blockquote>
  <p>I am using user agent:
  <code>private final static String USER_AGENT = &quot;Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) </code></p>
  <p>I've also tried to use different user agents, even mobile ones but I always get the same error, is it related to Javascript maybe?</p>
  <p>My code is the following, if needed:</p>
  <pre><code>public void getPageName(String urlString) throws Exception {
          try (final WebClient webClient = new WebClient(BrowserVersion.BEST_SUPPORTED)) {
              boolean javascriptEnabled = true;
  
              webClient.setRefreshHandler(new WaitingRefreshHandler(TIMEOUT / 1000));
              webClient.setCssErrorHandler(new SilentCssErrorHandler());
              webClient.setJavaScriptErrorListener(new SilentJavaScriptErrorListener());
              webClient.getCache().setMaxSize(0);
  
              webClient.getOptions().setRedirectEnabled(true);
              webClient.getOptions().setUseInsecureSSL(true);
              webClient.getOptions().setJavaScriptEnabled(javascriptEnabled);
              webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
              webClient.getOptions().setThrowExceptionOnScriptError(false);
              webClient.getOptions().setTimeout(TIMEOUT);
      
      //Boilerpipe // NOTE: Use ArticleExtractor unless DefaultExtractor gives better results for you
              URL  url = new URL(urlString);
              InputSource is = new InputSource();
              is.setEncoding(&quot;UTF-8&quot;);
              is.setByteStream(url.openStream());
              String text = DefaultExtractor.INSTANCE.getText(is);
  
  
              System.out.println(&quot;\n******************\n&quot;);
              System.out.println(text);
              System.out.println(&quot;\n******************\n&quot;);
  
              writeIntoFile(text);
  
          }
          catch (Exception e){
              System.out.println(&quot;Error when reading page  &quot; + e);
          }
      }
  
  </code></pre>