title: Apache HTTPClient throws java.net.SocketException: Connection reset for many domains
body: <p>I'm creating a (well behaved) web spider and I notice that some servers are causing Apache HttpClient to give me a SocketException -- specifically:</p>
  
  <pre><code>java.net.SocketException: Connection reset
  </code></pre>
  
  <p>The code that causes this is:</p>
  
  <pre><code>// Execute the request
  HttpResponse response; 
  try {
      response = httpclient.execute(httpget); //httpclient is of type HttpClient
  } catch (NullPointerException e) {
      return;//deep down in apache http sometimes throws a null pointer...  
  }
  </code></pre>
  
  <p>For most servers it's just fine.  But for others, it immediately throws a SocketException.</p>
  
  <p>Example of site that causes immediate SocketException: <a href="http://www.bhphotovideo.com/" rel="noreferrer">http://www.bhphotovideo.com/</a></p>
  
  <p>Works great (as do most websites): <a href="http://www.google.com/" rel="noreferrer">http://www.google.com/</a></p>
  
  <p>Now, as you can see, www.bhphotovideo.com loads fine in a web browser.  It also loads fine when I don't use Apache's HTTP Client.  (Code like this:)</p>
  
  <pre><code> HttpURLConnection c = (HttpURLConnection)url.openConnection();  
   BufferedInputStream in = new BufferedInputStream(c.getInputStream());  
   Reader r = new InputStreamReader(in);     
  
   int i;  
   while ((i = r.read()) != -1) {  
        source.append((char) i);  
   }  
  </code></pre>
  
  <p>So, why don't I just use this code instead?  Well there are some key features in Apache's HTTP Client that I need to use.</p>
  
  <p>Does anyone know what causes some servers to cause this exception?</p>
  
  <p>Research so far:</p>
  
  <ul>
  <li><p>Problem occurs on my local Mac dev machines AND an AWS EC2 Instance, so it's not a local firewall.</p></li>
  <li><p>It seems the error isn't caused by the remote machine because the exception doesn't say "by peer"</p></li>
  <li><p>This stack overflow seems relavent <a href="https://stackoverflow.com/questions/62929/java-net-socketexception-connection-reset">java.net.SocketException: Connection reset</a> but the answers don't show why this would happen only from Apache HTTP Client and not other approaches.</p></li>
  </ul>
  
  <p>Bonus question: I'm doing a fair amount of crawling with this system.  Is there generally a better Java class for this other than Apache HTTP Client?  I've found a number of issues (such as the NullPointerException I have to catch in the code above).  It seems that HTTPClient is very picky about server communications -- more picky than I'd like for a crawler that can't just break when a server doesn't behave.</p>
  
  <p>Thanks all! </p>
  
  <h1>Solution</h1>
  
  <p>Honestly, I don't have a perfect solution, but it works, so that's good enough for me.</p>
  
  <p>As pointed out by oleg below, Bixo has created a crawler that customizes HttpClient to be more forgiving to servers.  To "get around" the issue more than fix it, I just used SimpleHttpFetcher provided by Bixo here:
  (linked removed - SO thinks I'm a spammer, so you'll have to google it yourself)</p>
  
  <pre><code>SimpleHttpFetcher fetch = new SimpleHttpFetcher(new UserAgent("botname","contact@yourcompany.com","ENTER URL"));
  try {
      FetchedResult result = fetch.fetch("ENTER URL");
      System.out.println(new String(result.getContent()));
  } catch (BaseFetchException e) {
      e.printStackTrace();
  }
  </code></pre>
  
  <p>The down side to this solution is that there are a lot of dependencies for Bixo -- so this may not be a good work around for everyone.  However, you can always just work through their use of DefaultHttpClient and see how they instantiated it to get it to work.  I decided to use the whole class because it handles some things for me, like automatic redirect following (and reporting the final destination url) that are helpful.</p>
  
  <p>Thanks for the help all.</p>
  
  <h1>Edit: TinyBixo</h1>
  
  <p>Hi all.  So, I loved how Bixo worked, but didn't like that it had so many dependencies (including all of Hadoop).  So, I created a vastly simplified Bixo, without all the dependencies.  If you're running into the problems above, I would recommend using it (and feel free to make pull requests if you'd like to update it!)</p>
  
  <p>It's available here: <a href="https://github.com/juliuss/TinyBixo" rel="noreferrer">https://github.com/juliuss/TinyBixo</a> </p>