title: Upload a large XML file with Python Requests library
body: <p>I'm trying to replace curl with Python &amp; the requests library. With curl, I can upload a single XML file to a REST server with the curl -T option. I have been unable to do the same with the requests library.</p>
  
  <p>A basic scenario works:</p>
  
  <pre><code>payload = '&lt;person test="10"&gt;&lt;first&gt;Carl&lt;/first&gt;&lt;last&gt;Sagan&lt;/last&gt;&lt;/person&gt;'
  headers = {'content-type': 'application/xml'}
  r = requests.put(url, data=payload, headers=headers, auth=HTTPDigestAuth("*", "*"))
  </code></pre>
  
  <p>When I change payload to a bigger string by opening an XML file, the .put method hangs (I use the codecs library to get a proper unicode string). For example, with a 66KB file:</p>
  
  <pre><code>xmlfile = codecs.open('trb-1996-219.xml', 'r', 'utf-8')
  headers = {'content-type': 'application/xml'}
  content = xmlfile.read()
  r = requests.put(url, data=content, headers=headers, auth=HTTPDigestAuth("*", "*"))
  </code></pre>
  
  <p>I've been looking into using the multipart option (files), but the server doesn't seem to like that.</p>
  
  <p>So I was wondering if there is a way to simulate curl -T behaviour in Python requests library.</p>
  
  <p>UPDATE 1: 
  The program hangs in textmate, but throws an UnicodeEncodeError error on the commandline. Seems that must be the problem. So the question would be: is there a way to send unicode strings to a server with the requests library? </p>
  
  <p>UPDATE 2:
  Thanks to the comment of Martijn Pieters the UnicodeEncodeError went away, but a new issue turned up.
  With a literal (ASCII) XML string, logging shows the following lines:</p>
  
  <pre><code>2012-11-11 15:55:05,154 INFO Starting new HTTP connection (1): my.ip.address
  2012-11-11 15:55:05,294 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 401 211
  2012-11-11 15:55:05,430 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 201 0
  </code></pre>
  
  <p>Seems the server always bounces the first authentication attempt (?) but then accepts the second one.</p>
  
  <p>With a file object (open('trb-1996-219.xml', 'rb')) passed to data, the logfile shows:</p>
  
  <pre><code>2012-11-11 15:50:54,309 INFO Starting new HTTP connection (1): my.ip.address
  2012-11-11 15:50:55,105 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 401 211
  2012-11-11 15:51:25,603 WARNING Retrying (0 attempts remain) after connection broken by 'BadStatusLine("''",)': /v1/documents?uri=/example/test.xml
  </code></pre>
  
  <p>So, first attempt is blocked as before, but no second attempt is made. </p>
  
  <p>According to Martijn Pieters (below), the second issue can be explained by a faulty server (empty line). 
  I will look into this, but if someone has a workaround (apart from using curl) I wouldn't mind hearing it. </p>
  
  <p>And I am still surprised that the requests library behaves so differently for small string and file object. Isn't the file object serialized before it gets to the server anyway?</p>