title: libcurl compress downloaded data on the fly
body: <p>Linux. I'm downloading a lot of data via libcurl. A data has low entropy but huge size (Tens of Gb). At the moment I first download file and then compress it with zlib.<br>
  But this takes more time and more space. So I'm trying to implement compress on the fly in libcurl write callback.<br>
  The main problem is exact size of data to download is unknown. The code is dirty but its just a test. And it seems this is not a way to go. Maybe some other way is more suitable?</p>
  
  <pre><code>unsigned char *outZ=malloc(1500);//Maximum write chunk is 1448
  ...
  size_t
  curl_write_func(void *ptr, size_t size, size_t nmemb, void *stream)
  {   
      size_t data=zip(ptr,size*nmemb,outZ,1500,Z_FINISH);    
      fwrite(outZ, 1, data, (FILE *) stream); 
      return (size*nmemb);
  }
  
  size_t
  zip(void *source, size_t src_size, void *target,size_t tgt_size, int mode)
  {   
      z_stream strm;          
      strm.zalloc = Z_NULL;
      strm.zfree = Z_NULL;
      strm.opaque = Z_NULL;
      deflateInit(&amp;strm, Z_BEST_COMPRESSION);
      strm.next_in = source;
      strm.avail_in = src_size;
      strm.next_out=target;
      strm.avail_out=tgt_size;
      deflate(&amp;strm, mode);
      deflateEnd(&amp;strm);
      return (tgt_size - strm.avail_out);
  }
  </code></pre>