title: URL.openStream() is very slow when ran on school's unix server
body: <p>I am using <code>URL.openStream()</code> to download many html pages for a crawler that I am writing.  The method runs great locally on my mac however on my schools unix server the method is extremely slow.  <em>But only when downloading the first page.</em>  </p>
  
  <p>Here is the method that downloads the page:</p>
  
  <pre><code>public static String download(URL url) throws IOException {
      Long start = System.currentTimeMillis();
      InputStream is = url.openStream();
      System.out.println("\t\tCreated 'is' in "+((System.currentTimeMillis()-start)/(1000.0*60))+"minutes");
      ...
  }
  </code></pre>
  
  <p>And the main method that invokes it: </p>
  
  <pre><code>LinkedList&lt;URL&gt; ll = new LinkedList&lt;URL&gt;();
  ll.add(new URL("http://sheldonbrown.org/bicycle.html"));
  ll.add(new URL("http://www.trentobike.org/nongeo/index.html"));
  ll.add(new URL("http://www.trentobike.org/byauthor/index.html"));
  ll.add(new URL("http://www.myra-simon.com/bike/travel/index.html"));
  for (URL tmp : ll) {
      System.out.println();
      System.out.println(tmp);
      CrawlerTools.download(tmp);
  }
  </code></pre>
  
  <p>Output locally <em>(Note: all are fast)</em>: </p>
  
  <blockquote>
    <p><a href="http://sheldonbrown.org/bicycle.html" rel="nofollow">http://sheldonbrown.org/bicycle.html</a>
                    Created 'is' in 0.00475minutes</p>
    
    <p><a href="http://www.trentobike.org/nongeo/index.html" rel="nofollow">http://www.trentobike.org/nongeo/index.html</a>
                    Created 'is' in 0.005083333333333333minutes</p>
    
    <p><a href="http://www.trentobike.org/byauthor/index.html" rel="nofollow">http://www.trentobike.org/byauthor/index.html</a>
                    Created 'is' in 0.0023833333333333332minutes</p>
    
    <p><a href="http://www.myra-simon.com/bike/travel/index.html" rel="nofollow">http://www.myra-simon.com/bike/travel/index.html</a>
                    Created 'is' in 0.00405minutes</p>
  </blockquote>
  
  <p>Output on School Machine Server <em>(Note: All are fast except the first one.  The first one is slow regardless of what the first site is)</em>:</p>
  
  <blockquote>
    <p><a href="http://sheldonbrown.org/bicycle.html" rel="nofollow">http://sheldonbrown.org/bicycle.html</a>
                    Created 'is' in 3.2330666666666668minutes</p>
    
    <p><a href="http://www.trentobike.org/nongeo/index.html" rel="nofollow">http://www.trentobike.org/nongeo/index.html</a>
                    Created 'is' in 0.016416666666666666minutes</p>
    
    <p><a href="http://www.trentobike.org/byauthor/index.html" rel="nofollow">http://www.trentobike.org/byauthor/index.html</a>
                    Created 'is' in 0.0022166666666666667minutes</p>
    
    <p><a href="http://www.myra-simon.com/bike/travel/index.html" rel="nofollow">http://www.myra-simon.com/bike/travel/index.html</a>
                    Created 'is' in 0.009533333333333333minutes</p>
  </blockquote>
  
  <p>I am not sure if this is a Java issue (*A problem in my Java code) or a server issue.  What are my options?  </p>
  
  <hr>
  
  <p>When run on the server this is the output of the time command: </p>
  
  <pre><code>real    3m11.385s
  user    0m0.277s
  sys     0m0.113s
  </code></pre>
  
  <p>I am not sure if this is relevant...  What should I do to try and isolate my problem..?</p>