title: Missing bytes when uploading zip file with multipart/form-data using java
body: <p>I'm having an issue with uploading a .zip file to a remote server in that some bytes are missing from the file after the upload. Upon redownloading the file, the .zip archive is unopenable, which leads me to believe that I need those bytes to perform the upload successfully.</p>
  
  <p>I am using a multipart/form-data POST request to upload the file. The utility helper class I use to do this is given in the below code:</p>
  
  <pre><code>import java.io.*;
  import java.net.HttpURLConnection;
  import java.net.URL;
  import java.net.URLConnection;
  import java.util.ArrayList;
  import java.util.LinkedHashMap;
  import java.util.List;
  
  public class MultipartFormDataUtil {
      private final String boundary;
      private static final String lineReturn = "\r\n";
      private HttpURLConnection conn;
      private DataOutputStream dos;
      int bytesRead, bytesAvail, bufferSize;
      byte[] buffer;
      int maxBufferSize = 1*1024*1024;
      List&lt;String&gt; response;
  
      public MultipartFormDataUtil(String postUrl, LinkedHashMap&lt;String, String&gt; params, File file) throws IOException {
          boundary = "=-=" + System.currentTimeMillis() + "=-=";
  
          URL url = new URL(postUrl);
          conn = (HttpURLConnection) url.openConnection();
          conn.setDoInput(true);
          conn.setDoOutput(true);
          conn.setUseCaches(false);
          conn.setRequestMethod("POST");
          conn.setRequestProperty("Connection", "Keep-Alive");
          conn.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
          dos = new DataOutputStream(conn.getOutputStream());
  
          for (String key : params.keySet()) {
              addFormPart(key, params.get(key));
          }
  
          addFilePart(file);
  
          finish();
      }
  
      private void addFormPart(String name, String value) throws IOException {
          dos.writeBytes("--" + boundary + lineReturn);
          dos.writeBytes("Content-Disposition: form-data; name=\"" + name + "\"" + lineReturn);
          dos.writeBytes("Content-Type: text/plain" + lineReturn + lineReturn);
          dos.writeBytes(value + lineReturn);
          dos.flush();
      }
  
      private void addFilePart(File file) throws IOException {
          FileInputStream fileInputStream = new FileInputStream(file);
  
          dos.writeBytes("--" + boundary + lineReturn);
          dos.writeBytes("Content-Disposition: form-data; name=\"file\"; filename=\"" + file.getName() + "\"" + lineReturn);
          dos.writeBytes("Content-Type: " + URLConnection.guessContentTypeFromName(file.getName()) + lineReturn);
          dos.writeBytes("Content-Transfer-Encoding: binary" + lineReturn + lineReturn);
  
          bytesAvail = fileInputStream.available();
          bufferSize = Math.min(bytesAvail, maxBufferSize);
          buffer = new byte[bufferSize];
  
          bytesRead = fileInputStream.read(buffer, 0, bufferSize);
  
          while (bytesRead &gt; 0) {
              dos.write(buffer, 0, bufferSize);
              bytesAvail = fileInputStream.available();
              bufferSize = Math.min(bytesAvail, maxBufferSize);
              bytesRead = fileInputStream.read(buffer, 0, bufferSize);
          }
          dos.flush();
  
          dos.writeBytes(lineReturn);
          dos.flush();
          fileInputStream.close();
      }
  
      private void finish() throws IOException {
          response = new ArrayList&lt;String&gt;();
  
          dos.writeBytes("--" + boundary + "--" + lineReturn);
          dos.flush();
          dos.close();
  
          BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
          String line;
  
          while ((line = reader.readLine()) != null) {
              response.add(line);
          }
  
          reader.close();
          conn.disconnect();
      }
  
      public List&lt;String&gt; getResponse() {
          return response;
      }
  </code></pre>
  
  <p>To give credit where credit is due, this utility is based off of examples from <a href="http://petersnotes.blogspot.com/2012/03/sending-multipart-form-data-from-java.html" rel="nofollow">Peter's Notes</a> and <a href="http://www.codejava.net/java-se/networking/upload-files-by-sending-multipart-request-programmatically" rel="nofollow">CodeJava.net</a>. This util is called with the following code:</p>
  
  <pre><code>protected static void postFile(String url, LinkedHashMap&lt;String, String&gt; params, File file) throws Exception {
      try {
          MultipartFormDataUtil multipartRequest = new MultipartFormDataUtil(url, params, file);
          List&lt;String&gt; response = multipartRequest.getResponse();
  
          for (String line : response) {
              System.out.println(line);
          }
  
      } catch (IOException ioe) {
          log.warn("There was an error posting the file and form data", ioe);
      }
  }
  </code></pre>
  
  <p>The upload url in this case is to an Amazon S3 bucket, which passes it on to the destination system. It is at this final destination that I can see that the process that is supposed to be running on the .zip file has failed (note: the process is run by a Rails app and gives the error "Error identifying package type: can't dup NilClass"). Upon downloading the file, I see that the file size is 3,110,416 bytes instead of 3,110,466 bytes. I can no longer extract the archive to see what is in it; the mac archive utility responds with "Error 2 - No such file or directory".</p>
  
  <p>I lack the conceptual background in this area to get a feel for where in the process things may be going wrong. I am hoping that someone will be able to tell me that I made an error in the utility class, or let me know that something else is the problem.</p>
  
  <p>Thank you for any insight you can provide, and let me know if I can post anything else that would be of help.</p>
  
  <p>EDIT: Some additional information I gathered about different sizes of file uploads (in bytes):</p>
  
  <p>Original----------Uploaded----------Difference</p>
  
  <p>10,167,389______10,167,238______151</p>
  
  <p>3,110,466_______3,110,416_______50</p>
  
  <p>156,885_________156,885_________0</p>
  
  <p>95,639,352______95,637,925______1,427</p>
  
  <p>For the 3 files that had bytes missing following the upload, the % of total data lost was around (but not exactly) 0.0015% for each one, but not equal to each other.</p>