title: download large file from Jetty (ambari webhdfs) is slow
body: <p>I have a file about 5G, download from hdfs using python client at 12M/s, buy my network could reach 500M/s, and smaller file work fine. Then I reproduced this problem with curl. </p>
  
  <p>Here is curl debug log:</p>
  
  <p>curl -v -X GET <a href="http://x.x.x.x/file" rel="nofollow noreferrer">http://x.x.x.x/file</a></p>
  
  <pre><code>&gt; GET /webhdfs/v1/user/sohuvideo/online/srcFile/188/718/188718791/dat1_188718791_2020_4_11_17_4_172647e6e60.mp4?op=OPEN&amp;user.name=sohuvideo&amp;namenoderpcaddress=sotocyon&amp;offset=0 HTTP/1.1
  &gt; User-Agent: curl/7.29.0
  &gt; Host: x.x.x.com:50075
  &gt; Accept: */*
  &gt; 
  &lt; HTTP/1.1 200 OK
  &lt; Cache-Control: no-cache
  &lt; Expires: Tue, 21 Apr 2020 03:01:26 GMT
  &lt; Date: Tue, 21 Apr 2020 03:01:26 GMT
  &lt; Pragma: no-cache
  &lt; Expires: Tue, 21 Apr 2020 03:01:26 GMT
  &lt; Date: Tue, 21 Apr 2020 03:01:26 GMT
  &lt; Pragma: no-cache
  &lt; Content-Type: application/octet-stream
  &lt; Access-Control-Allow-Methods: GET
  &lt; Access-Control-Allow-Origin: *
  &lt; Transfer-Encoding: chunked
  &lt; Server: Jetty(6.1.26)
  &lt; 
  { [data not shown]
  100  119M    0  119M    0     0  13.0M      0 --:--:--  0:00:09 --:--:-- 12.1M^C
  </code></pre>
  
  <p>After some digging, I found if attach header <strong>Connection: close</strong> to the request, it could end up much faster.</p>
  
  <p>curl -v -H "Connection: close" -X  GET <a href="http://x.x.x.x/file" rel="nofollow noreferrer">http://x.x.x.x/file</a></p>
  
  <pre><code>&gt; GET /webhdfs/v1/user/sohuvideo/online/srcFile/188/718/188718791/dat1_188718791_2020_4_11_17_4_172647e6e60.mp4?op=OPEN&amp;user.name=sohuvideo&amp;namenoderpcaddress=sotocyon&amp;offset=0 HTTP/1.1
  &gt; User-Agent: curl/7.29.0
  &gt; Host: x.x.x.com:50075
  &gt; Accept: */*
  &gt; Connection: close
  &gt; 
  &lt; HTTP/1.1 200 OK
  &lt; Cache-Control: no-cache
  &lt; Expires: Tue, 21 Apr 2020 03:00:13 GMT
  &lt; Date: Tue, 21 Apr 2020 03:00:13 GMT
  &lt; Pragma: no-cache
  &lt; Expires: Tue, 21 Apr 2020 03:00:13 GMT
  &lt; Date: Tue, 21 Apr 2020 03:00:13 GMT
  &lt; Pragma: no-cache
  &lt; Content-Type: application/octet-stream
  &lt; Access-Control-Allow-Methods: GET
  &lt; Access-Control-Allow-Origin: *
  &lt; Connection: close
  &lt; Server: Jetty(6.1.26)
  &lt; 
  { [data not shown]
  100 4517M    0 4517M    0     0   138M      0 --:--:--  0:00:32 --:--:--  153M
  * Closing connection 0
  </code></pre>
  
  <p>I think this probably caused by <strong>Transfer-Encoding: chunked</strong> from server when file is large, server chose this because when server transfer the file the file size has not yet be decided, chunked stream could give a lots of overhead. If given <strong>Connection: close</strong> then server would not use <strong>Transfer-Encoding: chunked</strong> to indicate the end of steam, just close the connection instead.</p>
  
  <p>Is there any way to fix this from server side?</p>