title: How to open many HttpURLConnection?
body: <p>I'm trying to implement a simple URL availability checker which basically checks if the link is available (No HTTP 403, 404 etc. returned).</p>
  
  <p>I have more than 20,000 links(to different servers/websites) in my database for testing purposes but it doesn't seems to work when I try to create more than 10 Threads.</p>
  
  <p>Here is the code I'm using for opening the connection and reading response code within each WorkerThread.</p>
  
  <pre><code>                    URL url = new URL(dto.getUrl());
  
                      httpUrlConnection = (HttpURLConnection) url.openConnection();
                      httpUrlConnection.setUseCaches(false);
                      // httpUrlConnection.setConnectTimeout(6000);
                      httpUrlConnection.setDoInput(true);
                      httpUrlConnection.setDoOutput(false);
                      httpUrlConnection.setRequestMethod("GET");
                      httpUrlConnection.setRequestProperty("Host", dto.getUrl().replace("http://", ""));
                      // httpUrlConnection.setRequestProperty("Connection",
                      // "Keep-Alive");
                      httpUrlConnection.setRequestProperty("User-Agent", USER_AGENT);
                      httpUrlConnection.setRequestProperty("Cache-Control", "no-cache");
  
                      httpUrlConnection.connect();
  
                      int code = httpUrlConnection.getResponseCode();
  </code></pre>
  
  <p>Few issues I have noticed when having multiple threads opening the connections:</p>
  
  <p>1) Only first 100-200 connections seems to open with no problem, after that, I start getting "Read timeout", "Connection timeout", "Connection reset" etc. Although, if you try to run the code again the links which have thrown above exceptions will return proper response code (if they get processed in first 100).</p>
  
  <p>2) The response code sometimes is not valid (especially if the link was processed after first 100 links). I have noticed that sometimes 404 is returned when in fact it should return 200 (I checked it by putting link in first 100).</p>
  
  <p>I did try using Apache's Http client but it also fails to process links correctly with many threads.</p>
  
  <p>So does anyone know a solution to this problem ? What is the maximum amount of connections you could open using HttpURLConnection using multiple threads ? Is there any other way to open many HTTP connections and check response codes ?</p>
  
  <p>Thank you all in advance !</p>