title: perl script to access sec edgar master files returns file not found when file exists on ftp server
body: <p>It would be great if someone can help. I really am stuck.</p>
  
  <p>I am downloading the master files from SEC edgar and I got the script from—<a href="http://brage.bibsys.no/bi/bitstream/URN:NBN:no-bibsys_brage_38213/1/Norli_SRFE_2012.pdf" rel="nofollow">http://brage.bibsys.no/bi/bitstream/URN:NBN:no-bibsys_brage_38213/1/Norli_SRFE_2012.pdf</a> (page 14..published now)</p>
  
  <p>I get the error 404 master.gz not found
  While debugging i made it paste the url and when i use the same in browser I can download the file. It is parsing the url correctly till QTR1 but after that it is not able to find the file when it actually exists ..please help. </p>
  
  <p>1) for debugging reasons now I changed the code to 1995 (but later plan to add years 1995 to 2012)</p>
  
  <p>2) It did not work for any file. When I said QTR1 abovr - I meant that the same code without the file name (just for testing ) -- ....full-index/1995/QTR1/ (without the file name) returns a status code OK but ...ftp.sec.gov/edgar/full-index/1995/QTR1/master.gz returns 404 file not found error. It does not work for any quarter. </p>
  
  <p>I wasted so much time on this seemingly simple thing which is supposed to work but it is just not working.... could you copy past this and run..is it working for you? </p>
  
  <p>The code below gets the master files from QTR folders. Pasting my code ::</p>
  
  <p>—————-</p>
  
  <pre><code>use LWP::UserAgent;
  my $ua = LWP::UserAgent-&gt;new;
  $ua-&gt;timeout(250);
  $ua-&gt;env_proxy;
  
  for($year=1995; $year&lt;1996; $year=$year+1){
  for($i=1; $i&lt;5; $i=$i+1){
  $quarter = “QTR” . $i;
  $filegrag = “ftp://ftp.sec.gov/edgar/full-index/” . $year . “/” . $quarter .       “/master.gz”;
  print $filegrag;
  
  # This command gets the file from EDGAR
  my $response = $ua-&gt;get($filegrag);
  print $response;
    print $response-&gt;status_line;
  
  # Now just pipe the output to a file named appropriately
  $filename = $year . $quarter . “master”;
  open(MYOUTFILE, “&gt;” . $filename);
  if ($response-&gt;is_success)  {
  print MYOUTFILE $response-&gt;decoded_content;
  }
  else {
  die $response-&gt;status_line;
  }
  close(MYOUTFILE);
  }
  }
  </code></pre>