title: Perl beats Python in fetching HTML pages?
body: <p>I have a Perl script which is fetching html pages. I tried rewriting it in Python (Just Coz I am trying to learn Python) and I found it to be really slow!</p>
  
  <p>Here's the test Script in Perl</p>
  
  <pre><code>#!/usr/bin/perl
  use LWP::Simple;
  
  $url = "http://majorgeeks.com/page.php?id=";
  
  open(WEB,"&gt;&gt;"."perldata.txt");
  
  for ($column = 1 ; $column &lt;= 20 ; $column ++)
  
  {
  
      $temp = $url.$column;
      print "val = $temp\n\n";
  
      $response=get($temp)or die("[-] Failed!!\n");
  
      print WEB "$response\n\n";
  
  }
  </code></pre>
  
  <p>And here's the Equivalent Code in Python</p>
  
  <pre><code>import urllib2
  
  url = "http://majorgeeks.com/page.php?id="
  
  f = open("pydata.txt", 'w')
  
  for i in range(20):
  
     tempurl = url + str(i+1)
     print "Val : " + tempurl + "\n\n"
  
     #req = urllib2.Request(tempurl)
     res = urllib2.urlopen(tempurl)
  
     f.write(res.read())
  
  f.close()
  </code></pre>
  
  <p>The difference I found is huge!
  Perl Script finished in approx 30 seconds.
  While Python Script took 7mins approx. (420 seconds)!!</p>
  
  <p>I am using Ubuntu 11.10, 64bit, Core i7, Tested it on a 12MBPS connection. I tried this several times, and every time I get the same amount of difference.</p>
  
  <p>Am I doing something wrong here? Or I need to do something? Or the difference is justified? (I hope not)</p>
  
  <p>Thanks a lot for your help.</p>
  
  <p>Update 3:
  I just came home and booted my laptop, ran the code again and it finished it in 11 seconds!!! :/ Was it because I "rebooted" my comp??
  <a href="https://gist.github.com/1574040#file_gistfile1.py" rel="nofollow">Here's</a> the Profiler output</p>
  
  <p>Note - Perl still took 31 seconds for the same thing!! :/</p>
  
  <p>Update 2:
  As suggested by @Makoto <a href="https://gist.github.com/1572718#file_gistfile1.txt" rel="nofollow">Here's</a> the profiler data That I did. And It is is really slow! I know some python configuration has to do with this, but don't know what. For one simple request, It shouldn't take 20 seconds!!!</p>
  
  <p>Update : 
  Fixed url to tempurl. Commented out the urllib2.Request as suggested here. Not much difference at all.</p>