title: How can I Read and Transfer chunks of file with Hadoop WebHDFS?
body: <p>I need to transfer big files (at least 14MB) from the Cosmos instance of the FIWARE Lab to my backend.</p>
  
  <p>I used the Spring RestTemplate as a client interface for the Hadoop WebHDFS REST API described <a href="https://hadoop.apache.org/docs/r0.23.11/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Open_and_Read_a_File" rel="nofollow">here</a> but I run into an IO Exception:</p>
  
  <pre><code>Exception in thread "main" org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/&lt;user.name&gt;/&lt;path&gt;?op=open&amp;user.name=&lt;user.name&gt;":Truncated chunk ( expected size: 14744230; actual size: 11285103); nested exception is org.apache.http.TruncatedChunkException: Truncated chunk ( expected size: 14744230; actual size: 11285103)
      at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:580)
      at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:545)
      at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:466)
  </code></pre>
  
  <p>This is the actual code that generates the Exception:</p>
  
  <pre><code>RestTemplate restTemplate = new RestTemplate();
  restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory());
  restTemplate.getMessageConverters().add(new ByteArrayHttpMessageConverter()); 
  HttpEntity&lt;?&gt; entity = new HttpEntity&lt;&gt;(headers);
  
  UriComponentsBuilder builder = 
      UriComponentsBuilder.fromHttpUrl(hdfs_path)
          .queryParam("op", "OPEN")
          .queryParam("user.name", user_name);
  
  ResponseEntity&lt;byte[]&gt; response =
      restTemplate
          .exchange(builder.build().encode().toUri(), HttpMethod.GET, entity, byte[].class);
  
  FileOutputStream output = new FileOutputStream(new File(local_path));
  IOUtils.write(response.getBody(), output);
  output.close();
  </code></pre>
  
  <p>I think this is due to a transfer timeout on the Cosmos instance, so I tried to 
  send a <code>curl</code> on the path by specifying <code>offset, buffer and length</code> parameters, but they seem to be ignored: I got the whole file.</p>
  
  <p>Thanks in advance.</p>