title: How to determine optimal number of threads for high latency network requests?
body: <p>I am writing a utility that must make thousands of network requests. Each request receives only a single, small packet in response (similar to ping), but may take upwards of several seconds to complete. Processing each response completes in one (simple) line of code.</p>
  
  <p>The net effect of this is that the computer is not IO-bound, file-system-bound, or CPU-bound, it is only bound by the latency of the responses.</p>
  
  <p>This is similar to, but <em>not</em> the same as <a href="https://stackoverflow.com/questions/6065374/there-is-a-way-to-determine-the-ideal-number-of-threads">There is a way to determine the ideal number of threads?</a> and <a href="https://stackoverflow.com/questions/16137409/java-best-way-to-determine-the-optimal-number-of-threads">Java best way to determine the optimal number of threads [duplicate]</a>... the primary difference is that I am only bound by latency.</p>
  
  <p>I am using an <code>ExecutorService</code> object to run the threads and a <code>Queue&lt;Future&lt;Integer&gt;&gt;</code> to track threads that need to have results retrieved:</p>
  
  <pre><code>ExecutorService executorService = Executors.newFixedThreadPool(threadPoolSize);
  Queue&lt;Future&lt;Integer&gt;&gt; futures = new LinkedList&lt;Future&lt;Integer&gt;&gt;();
  
  for (int quad3 = 0 ; quad3 &lt; 256 ; ++quad3) {
      for (int quad4 = 0 ; quad4 &lt; 256 ; ++quad4) {
          byte[] quads = { quad1, quad2, (byte)quad3, (byte)quad4 };
          futures.add(executorService.submit(new RetrieverCallable(quads)));
      }
  }
  </code></pre>
  
  <p>... I then dequeue all the elements in the queue and put the results in the required data structure:</p>
  
  <pre><code>int[] result = int[65536]
  while(!futures.isEmpty()) {
      try {
          results[i] = futures.remove().get();
      } catch (Exception e) {
          addresses[i] = -1;
      }
  }
  </code></pre>
  
  <p>My first question is: Is this a reasonable way to track all the threads? If thread X takes a while to complete, many other threads might finish before X does. Will the thread pool exhaust itself waiting for open slots, or will the <code>ExecutorService</code> object manage the pool in such a way that threads that have completed but not yet been processed be moved out of available slots so that other threads my begin?</p>
  
  <p>My second question is what guidelines can I use for finding the optimal number of threads to make these calls? I don't even know order-of-magnitude guidance here. I know it works pretty well with 256 threads, but seems to take roughly the same overall time with 1024 threads. CPU utilization is hovering around 5%, so that doesn't appear to be an issue. With that large a number of threads, what are all the metrics I should be looking at to compare different numbers? Obviously overall time to process the batch, average time per thread... what else? Is memory an issue here?</p>