title: How to programatically retrieve file that is on a web page using java
body: <p>I am trying to download a file (which I believe is an Application/OCTET-STREAM attachment) from a web page.
  The URL looks like <a href="http://REDACTED.com/bb/avfile/filename" rel="nofollow">http://REDACTED.com/bb/avfile/filename</a>. When I enter this Url in a browser (IE or Firefox) the file downloads normally, with a size of 72KB, with no file extension and can be opened with my spreadsheet program . When I try to download the file through my java program it retrieves a file of only 70KB which will not open in my spreadsheet program. Trying to parse the file with  org.apache.poi.xssf.usermodel.* produces an error of " Package should contain a content type part [M1.13]" which makes sense to me as the file is smaller that the file downloading from the Browser</p>
  
  <p>It seems the server will only deliver the correct file if the URL request is comming from a browser. Here is my code: </p>
  
  <pre><code>String UrlString = "http://REDACTED.com/bb/avfile/filename";
  URL internetURL = new URL(UrlString);
  URLConnection internetConnection = internetURL.openConnection();
  
  destinationFile = new File(filePath + "test");
  org.apache.commons.io.FileUtils.copyURLToFile (internetURL, destinationFile,2000,2000);
  </code></pre>
  
  <p>I have tried casting to an httpConnection</p>
  
  <pre><code>HttpURLConnection httpConnection = (HttpURLConnection) new URL(UrlString).openConnection();
  httpConnection.setRequestMethod("GET");
  </code></pre>
  
  <p>I have tried setting the setRequestProperty methods to match the headers to what is shown in HTTPFOX when i submit the URL throught the browserlike so :</p>
  
  <pre><code>httpConnection.setRequestProperty ("Content-Type","application/vnd.ms-excel");
  httpConnection.setRequestProperty ("User-Agent","Mozilla/5.0 (Windows NT 5.1; rv:9.0.1) Gecko/20100101 Firefox/9.0.1");
  // etc,
  </code></pre>
  
  <p>Also getting the file as a stream :</p>
  
  <pre><code>destinationFile = new File(filePath + "temp2");
  org.apache.commons.io.FileUtils.copyInputStreamToFile
  (httpConnection.getInputStream(), destinationFile);
  </code></pre>
  
  <p>All methods produce a file that is 2KB smaller than the file downloaded from the browser. The only difference I can see is the headers from the browser as shown by HTTPFox shows the request line as "bb/avfile/filenameHTTP1.1" with the host being "REDACTED.com". Adding HTTP1.1 to the URL in either the browser or through the program, produces a 1K file that reads "av content requested is not available". HTTPFox shows that there are no POST requests</p>
  
  <p>I am at a loss as to how to proceed . Any Suggestions or ideas on what problem is ?  </p>