title: Response encoding with node.js "request" module
body: <p>I am trying to get data from the Bing search API, and since the existing libraries seem to be based on old discontinued APIs I though I'd try myself using the <code>request</code> library, which appears to be the most common library for this. 
  My code looks like</p>
  
  <pre><code>var SKEY           =  "myKey...." , 
      ServiceRootURL =  'https://api.datamarket.azure.com/Bing/Search/v1/Composite';
  
  function getBingData(query, top, skip, cb) {
      var params = {
           Sources: "'web'", 
           Query: "'"+query+"'", 
           '$format': "JSON", 
           '$top': top, '$skip': skip
         },
         req = request.get(ServiceRootURL).auth(SKEY, SKEY, false).qs(params);
      request(req, cb)
  }
  
  getBingData("bookline.hu", 50, 0, someCallbackWhichParsesTheBody)
  </code></pre>
  
  <p>Bing returns some JSON and I can work with it sometimes <strong>but</strong> if the response body contains a large amount of non ASCII characters <code>JSON.parse</code> complains that the string is malformed. I tried switching to an ATOM content type, but there was no difference, the xml was invalid. Inspecting the response body as available in the <code>request()</code> callback actually shows bad code. </p>
  
  <p>So I tried the same request with some python code, and that appears to work fine all the time. For reference:</p>
  
  <pre><code>r = requests.get(
         'https://api.datamarket.azure.com/Bing/Search/v1/Composite?Sources=%27web%27&amp;Query=%27sexy%20cosplay%20girls%27&amp;$format=json', 
          auth=HTTPBasicAuth(SKEY,SKEY))
  stuffWithResponse(r.json())
  </code></pre>
  
  <p>I am unable to reproduce the problem with smaller responses (e.g. limiting the number of results) and unable to identify a single result which causes the issue (by stepping up the offset). 
  My impression is that the response gets read in chunks, transcoded somehow and reassembled back in a bad way, which means the json/atom data becomes invalid if some multibyte character gets split, which happens on larger responses but not small ones. </p>
  
  <p>Being new to node, I am not sure if there is something I should be doing (setting the encoding somewhere? Bing returns UTF-8, so this doesn't seem needed). </p>
  
  <p>Anyone has any idea of what is going on?</p>
  
  <p>FWIW, I'm on OSX 10.8, node is v0.8.20 installed via macports, request is v2.14.0 installed via npm.</p>