title: Python Requests Emulate a CURL POST sending multipart request with 1 or more files AND JSON body
body: <p>I've been hacking at this for two days now with no luck!</p>
  
  <p>WORKING CURL request </p>
  
  <pre><code>curl -X POST -v "http://$1:8080/controller/endpoint" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" -F "message={ \"id\": \"b3562c86-6ff4-4bf7-9c4a-4c64fff4d0ea\", \"stuff\": [
  {
  \"id\": \"1ca2d9b1-1d73-432a-b483-be404afff8da\",
  .......
  \"endTime\": \"\"
  }]}};type=application/json" -F "files=@file.zip"
  </code></pre>
  
  <p>Returns output that looks like:</p>
  
  <pre><code> ./rest.sh http://127.0.0.1/anything
  * Hostname was NOT found in DNS cache
  *   Trying 127.0.0.1...
  * Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
  &gt; POST /anything HTTP/1.1
  &gt; User-Agent: curl/7.35.0
  &gt; Host: 127.0.0.1
  &gt; Accept: */*
  &gt; Cache-Control: no-cache
  &gt; Content-Length: 493
  &gt; Expect: 100-continue
  &gt; Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW; boundary=------------------------52912a6946761b42
  &gt;
  &lt; HTTP/1.1 100 Continue
  &lt; HTTP/1.1 200 OK
  * Server gunicorn/19.9.0 is not blacklisted
  &lt; Server: gunicorn/19.9.0
  &lt; Date: Tue, 12 Feb 2019 18:18:56 GMT
  &lt; Connection: keep-alive
  &lt; Content-Type: application/json
  &lt; Content-Length: 725
  &lt; Access-Control-Allow-Origin: *
  &lt; Access-Control-Allow-Credentials: true
  &lt;
  {
    "args": {},
    "data": "",
    "files": {
      "files": "ZIP-CONTENT-GOES-HERE"
    },
    "form": {
      "message": "{ \"runId\": \"1ca2d9b1-1d73-432a-b483-be404a13e8da\", \"reports\": [\n{\n\"executionId\": \"1ca2d9b1-1d73-432a-b483-be404a13e8da\",\n\"endTime\": \"\"\n}]}}"
    },
    "headers": {
      "Accept": "*/*",
      "Cache-Control": "no-cache",
      "Content-Length": "493",
      "Content-Type": "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW; boundary=------------------------52912a6946761b42",
      "Expect": "100-continue",
      "Host": "127.0.0.1",
      "User-Agent": "curl/7.35.0"
    },
    "json": null,
    "method": "POST",
    "origin": "172.17.42.1",
    "url": "http://127.0.0.1/anything"
  }
  * Connection #0 to host 127.0.0.1 left intact
  </code></pre>
  
  <p>Now, if I added <code>,scrub2.zip</code> to the curl command (sending 2 zip files, AND JSON data), I get output which looks as follows:</p>
  
  <pre><code> ./rest.sh http://127.0.0.1/anything
  * Hostname was NOT found in DNS cache
  *   Trying 127.0.0.1...
  * Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
  &gt; POST /anything HTTP/1.1
  &gt; User-Agent: curl/7.35.0
  &gt; Host: 127.0.0.1
  &gt; Accept: */*
  &gt; Cache-Control: no-cache
  &gt; Content-Length: 878
  &gt; Expect: 100-continue
  &gt; Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW; boundary=------------------------27d684afce904423
  &gt;
  &lt; HTTP/1.1 100 Continue
  &lt; HTTP/1.1 200 OK
  * Server gunicorn/19.9.0 is not blacklisted
  &lt; Server: gunicorn/19.9.0
  &lt; Date: Tue, 12 Feb 2019 18:20:36 GMT
  &lt; Connection: keep-alive
  &lt; Content-Type: application/json
  &lt; Content-Length: 1117
  &lt; Access-Control-Allow-Origin: *
  &lt; Access-Control-Allow-Credentials: true
  &lt;
  {
    "args": {},
    "data": "",
    "files": {},
    "form": {
      "files": "--------------------------fd702594c1765b85\r\nContent-Disposition: attachment; filename=\"scrubbed.zip\"\r\nContent-Type: application/octet-stream\r\n\r\nZIP-CONTENT-GOES-HERE\r\n--------------------------fd702594c1765b85\r\nContent-Disposition: attachment; filename=\"scrubbed2.zip\"\r\nContent-Type: application/octet-stream\r\n\r\nZIP-CONTENT-GOES-HERE222222222\n\r\n--------------------------fd702594c1765b85--",
      "message": "{ \"runId\": \"1ca2d9b1-1d73-432a-b483-be404a13e8da\", \"reports\": [\n{\n\"executionId\": \"1ca2d9b1-1d73-432a-b483-be404a13e8da\",\n\"endTime\": \"\"\n}]}}"
    },
    "headers": {
      "Accept": "*/*",
      "Cache-Control": "no-cache",
      "Content-Length": "878",
      "Content-Type": "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW; boundary=------------------------27d684afce904423",
      "Expect": "100-continue",
      "Host": "127.0.0.1",
      "User-Agent": "curl/7.35.0"
    },
    "json": null,
    "method": "POST",
    "origin": "172.17.42.1",
    "url": "http://127.0.0.1/anything"
  }
  * Connection #0 to host 127.0.0.1 left intact
  </code></pre>
  
  <p>Do you see the difference? The 2 files are now embedded inside the form/files instead of files and form/message showing separately!</p>
  
  <p>This sort of CURL requests is accepted on the Java API endpoint
  which looks as follows in debugger:
  <a href="https://i.stack.imgur.com/XLkrY.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XLkrY.png" alt="The way two files look in Java debugger when CURL sends"></a></p>
  
  <p>but all my attempts on Python, such as:</p>
  
  <pre><code>multipart_form_data_object = {
      'scrubbed.zip': (args.files[0], open(args.files[0], 'rb'), "application/json"),
      'files': (args.files[1], open(args.files[1], 'rb'), "application/json"),
      'message': (None, open(args.message, 'rb'), 'application/json')
  }
   response = requests.post(args.url + ':' + str(args.port) + '/' + args.endpoint, files=multipart_form_data_object,
                               proxies=proxies)
  </code></pre>
  
  <p>(Which is the closest I got it to work), looks like this:</p>
  
  <pre><code>multipart_form_data_object = {
      'scrubbed.zip': (args.files[0], open(args.files[0], 'rb'), "application/json"),
      'files': (args.files[1], open(args.files[1], 'rb'), "application/json"),
      'message': (None, open(args.message, 'rb'), 'application/json')
  }
  response = requests.post(args.url + ':' + str(args.port) + '/' + args.endpoint, files=multipart_form_data_object,
                           proxies=proxies)
  </code></pre>
  
  <p>Which outputs as:</p>
  
  <pre><code>{'Content-Length': '664', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.21.0', 'Connection': 'keep-alive', 'Content-Type': 'multipart/form-data; boundary=227d4ef5a41db8a690e5cebadf336851'}
  {
    "args": {},
    "data": "",
    "files": {
      "files": "ZIP-CONTENT-GOES-HERE",
      "scrubbed.zip": "ZIP-CONTENT-GOES-HERE22222"
    },
    "form": {
      "message": "{\r\n  \"runId\": \"9c4a-4c64f6d4d0ea\",\r\n  \"reports\": [\r\n    {\r\n      \"executionId\": \"d73-432a-b483-be404a13e8da\",\r\n      \"endTime\": \"\"\r\n    }\r\n  ]\r\n}"
    },
    "headers": {
      "Accept": "*/*",
      "Accept-Encoding": "gzip, deflate",
      "Connection": "keep-alive",
      "Content-Length": "664",
      "Content-Type": "multipart/form-data; boundary=227d4ef5a41db8a690e5cebadf336851",
      "Host": "java.api.host.com",
      "User-Agent": "python-requests/2.21.0"
    },
    "json": null,
    "method": "POST",
    "origin": "10.0.0.2",
    "url": "http://java.api.host.com/anything"
  }
  </code></pre>
  
  <p>Now, attempting to adjust this for sending an array of files (otherwise, if I were to rename <code>scrubbed.zip</code> to <code>files</code> , it gets overwritten), to make it look like:</p>
  
  <pre><code>multipart_form_data_object = {
      'files': [(args.files[0], open(args.files[0], 'rb'), "application/json"),
       (args.files[1], open(args.files[1], 'rb'), "application/json")],
      'message': (None, open(args.message, 'rb'), 'application/json')
  }
  </code></pre>
  
  <p>Causes the error:</p>
  
  <pre><code>Traceback (most recent call last):
    File ".\load_stress_test_endpoint.py", line 84, in &lt;module&gt;
      post()
    File ".\load_stress_test_endpoint.py", line 76, in post
      proxies=proxies)
    File "C:\Python\Python27\lib\site-packages\requests\api.py", line 116, in post
      return request('post', url, data=data, json=json, **kwargs)
    File "C:\Python\Python27\lib\site-packages\requests\api.py", line 60, in request
      return session.request(method=method, url=url, **kwargs)
    File "C:\Python\Python27\lib\site-packages\requests\sessions.py", line 519, in request
      prep = self.prepare_request(req)
    File "C:\Python\Python27\lib\site-packages\requests\sessions.py", line 462, in prepare_request
      hooks=merge_hooks(request.hooks, self.hooks),
    File "C:\Python\Python27\lib\site-packages\requests\models.py", line 316, in prepare
      self.prepare_body(data, files, json)
    File "C:\Python\Python27\lib\site-packages\requests\models.py", line 504, in prepare_body
      (body, content_type) = self._encode_files(files, data)
    File "C:\Python\Python27\lib\site-packages\requests\models.py", line 169, in _encode_files
      body, content_type = encode_multipart_formdata(new_fields)
    File "C:\Python\Python27\lib\site-packages\urllib3\filepost.py", line 90, in encode_multipart_formdata
      body.write(data)
  TypeError: 'tuple' does not have the buffer interface
  </code></pre>
  
  <p>My final attempt was a different data structure (list), as follows:</p>
  
  <pre><code>multiple_files_list = [
      ('files', (args.files[0], open(args.files[0], 'rb'), "application/json")),
      ('files', (args.files[1], open(args.files[1], 'rb'), "application/json")),
      ('message', None, open(args.message, 'rb'), 'application/json')
  ]
  </code></pre>
  
  <p>Results in the error:</p>
  
  <pre><code>Traceback (most recent call last):
    File ".\load_stress_test_endpoint.py", line 84, in &lt;module&gt;
      post()
    File ".\load_stress_test_endpoint.py", line 76, in post
      proxies=proxies)
    File "C:\Python\Python27\lib\site-packages\requests\api.py", line 116, in post
      return request('post', url, data=data, json=json, **kwargs)
    File "C:\Python\Python27\lib\site-packages\requests\api.py", line 60, in request
      return session.request(method=method, url=url, **kwargs)
    File "C:\Python\Python27\lib\site-packages\requests\sessions.py", line 519, in request
      prep = self.prepare_request(req)
    File "C:\Python\Python27\lib\site-packages\requests\sessions.py", line 462, in prepare_request
      hooks=merge_hooks(request.hooks, self.hooks),
    File "C:\Python\Python27\lib\site-packages\requests\models.py", line 316, in prepare
      self.prepare_body(data, files, json)
    File "C:\Python\Python27\lib\site-packages\requests\models.py", line 504, in prepare_body
      (body, content_type) = self._encode_files(files, data)
    File "C:\Python\Python27\lib\site-packages\requests\models.py", line 141, in _encode_files
      for (k, v) in files:
  ValueError: too many values to unpack
  </code></pre>
  
  <p>Can you please advice, how to make Python request package perform similiarly to the CURL request?</p>
  
  <p>Follows is how the Java endpoint is set up:</p>
  
  <pre><code>public Response index(@RequestPart("message") @Valid
                            final Message message,
                            @ApiParam(value = "Multipart File array of compressed archives (zip) ", required = true) @RequestPart("files") @Valid
                            final MultipartFile[] files)
  </code></pre>