title: The most reliable way of getting contents of a web page?
body: <p>I am trying to find out the most reliable way of getting the content of a web page with url as an input parameter In Java? </p>
  
  <p>Things I have tried are : 
   1. JSOUP
   2. HtmlUnit
   3. URL and URLConnection</p>
  
  <p>The problem with 1 &amp; 2 is they sometimes throw SocketTimeoutException or behaves unpredictably even if it is possible to get the contents of the page (even if robots.txt allows it). </p>
  
  <p>Using 3 I am not able to get the loadTime without using the subtraction of milliseconds. that is the biggest problem, which yields inaccurate results.  Also for getting the content I need to use Streams and read the content line by line. </p>
  
  <p>The current implementation uses Approach #2. Which has capability of LoadTime and contentType. </p>
  
  <p>For Every web page's content I need contentType, loadTime etc. </p>
  
  <p>Basically it is for a Link Validator project which validates entire website including css's background-image also images, js, html etc. so based on contentType I filter and parse only HTML's content. </p>
  
  <p>P.S. Improving timeout more than 9 seconds will make the link validation slow. so my current timeout is 9 seconds. </p>
  
  <p>I need help, as I want to make my link validation tool as reliable as possible.</p>