title: How do I decompress large files using Zstd-jni and Byte Buffers
body: <p>I am trying to decompress a lot of 40 MB+ files as I download them in parallel using ByteBuffers and Channels.  I am getting better throughput by using Channels than I do by using Streams and we need this to be a very high throughput system as we need to process 40 TB of files every day and this part of the process is currently the bottleneck.  The files are compressed with <a href="https://github.com/luben/zstd-jni" rel="nofollow noreferrer">zstd-jni</a>.  Zstd-jni has api's for decompressing byte buffers but I get an error when I use them.  How do I decompress a byte buffer at a time using zstd-jni?</p>
  
  <p>I found these examples in their tests, but unless I am missing something the examples using ByteBuffers seem to assume the entire input file fits in one ByteBuffer:
  <a href="https://github.com/luben/zstd-jni/blob/master/src/test/scala/Zstd.scala" rel="nofollow noreferrer">https://github.com/luben/zstd-jni/blob/master/src/test/scala/Zstd.scala</a></p>
  
  <p>Below is my code for compressing and decompressing files.  The compression code works great, but the decompression code then fails with an error of -70.</p>
  
  <pre><code>public static long compressFile(String inFile, String outFolder, ByteBuffer inBuffer, ByteBuffer compressedBuffer, int compressionLevel) throws IOException {
      File file = new File(inFile);
      File outFile = new File(outFolder, file.getName() + ".zs");
      long numBytes = 0l;
  
      try (RandomAccessFile inRaFile = new RandomAccessFile(file, "r");
          RandomAccessFile outRaFile = new RandomAccessFile(outFile, "rw");
                  FileChannel inChannel = inRaFile.getChannel();
                  FileChannel outChannel = outRaFile.getChannel()) {
          inBuffer.clear();
          while(inChannel.read(inBuffer) &gt; 0) {
              inBuffer.flip();
              compressedBuffer.clear();
  
              long compressedSize = Zstd.compressDirectByteBuffer(compressedBuffer, 0, compressedBuffer.capacity(), inBuffer, 0, inBuffer.limit(), compressionLevel);
              numBytes+=compressedSize;
              compressedBuffer.position((int)compressedSize);
              compressedBuffer.flip();
              outChannel.write(compressedBuffer);
              inBuffer.clear(); 
          }
      }
  
      return numBytes;
  }
  
  public static long decompressFile(String originalFilePath, String inFolder, ByteBuffer inBuffer, ByteBuffer decompressedBuffer) throws IOException {
      File outFile = new File(originalFilePath);
      File inFile = new File(inFolder, outFile.getName() + ".zs");
      outFile = new File(inFolder, outFile.getName());
  
      long numBytes = 0l;
  
      try (RandomAccessFile inRaFile = new RandomAccessFile(inFile, "r");
          RandomAccessFile outRaFile = new RandomAccessFile(outFile, "rw");
                  FileChannel inChannel = inRaFile.getChannel();
                  FileChannel outChannel = outRaFile.getChannel()) {
  
          inBuffer.clear();
  
          while(inChannel.read(inBuffer) &gt; 0) {
              inBuffer.flip();
              decompressedBuffer.clear();
              long compressedSize = Zstd.decompressDirectByteBuffer(decompressedBuffer, 0, decompressedBuffer.capacity(), inBuffer, 0, inBuffer.limit());
              System.out.println(Zstd.isError(compressedSize) + " " + compressedSize);
              numBytes+=compressedSize;
              decompressedBuffer.position((int)compressedSize);
              decompressedBuffer.flip();
              outChannel.write(decompressedBuffer);
              inBuffer.clear(); 
          }
      }
  
      return numBytes;
  }
  </code></pre>