title: Slow & memory leak when looping MD5 calculations
body: <p>I have asked a similar question here <a href="https://stackoverflow.com/questions/18244691/performance-issues-when-looping-an-md5-calculator-on-many-files">Performance issues when looping an MD5 calculator on many files</a>, but I realized that the problem is not the code, but the GC, so I needed a new page for the same problem, but with a different kind of solution.</p>
  
  <p>I'm running an MD5 calculation loop on 1000+ image files. The program tournes slow after about 200, and uses a lot of memory (free -> inactive). It uses as much memory I had free when I started it (2GB-5GB).</p>
  
  <p>I tried to use Java.io / FastMD5. This code is the latest test with Java.nio.
  The problem occurred on all of them, and even FastMD5, which is an external library, so the problem is not the way I calculate the MD5, apparently.</p>
  
  <p>The problem seems to be the fact that the code runs so fast that the GC doesn't have time to keep up with the objects. </p>
  
  <p><strong>Are there any optimizations to the GC or my code that can help me with my memory problem?</strong></p>
  
  <p><strong>EDIT</strong> Tried to use ThreadLocal. No go. I realized that the program is somehow caching the MD5s, because after a slow loop (with a memory leak) comes a fast loop with no memory leak. After I release the memory the program goes slow again (and then fast).</p>
  
  <pre><code>public static void main(String[] args) {
  
          File[] directory = new File("/Users/itaihay/Desktop/Library/Seattle 2010").listFiles();
  
      for(int i = 0; i &lt; directory.length;i++){
  
          System.out.println(Utils.toMD5(directory[i]));
  
          }
  }
  </code></pre>
  
  <p>Utils.toMD5():</p>
  
  <pre><code>  public class Utils {
  
  
  public static final ThreadLocal&lt;MessageDigest&gt; mdT = new ThreadLocal&lt;MessageDigest&gt;(){
     protected MessageDigest initialValue(){
          try {
              return MessageDigest.getInstance("MD5");
          } catch (NoSuchAlgorithmException e) {
              e.printStackTrace();  
          }
         return null;
     }
  };
  
  public static final ThreadLocal&lt;ByteBuffer&gt; bufferT = new ThreadLocal&lt;ByteBuffer&gt;(){
      protected ByteBuffer initialValue(){
              return ByteBuffer.allocate(32000);
      }
  };
  
  private static Utils util = new Utils();
  private static MessageDigest md;
  private static FileChannel fileChannel;
  private static ByteBuffer buffer = bufferT.get();
  
  private Utils() {
  
  //            md = MessageDigest.getInstance("MD5");
          md = mdT.get();
  
  }
  public static String toMD5(File file) throws NoSuchAlgorithmException, IOException {
  //        BufferedInputStream fis = new BufferedInputStream(new FileInputStream(file));
  
     RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r");
  
      fileChannel = randomAccessFile.getChannel();
  
      /*while (fileChannel.read(buffer) != -1) {
          buffer.flip();
          md.update(buffer);
          buffer.clear();
      }*/
  
      while (fileChannel.read(bufferT.get()) != -1) {
          bufferT.get().flip();
          md.update(bufferT.get());
          bufferT.get().clear();
      }
  
      byte[] mdbytes = md.digest();
  
      randomAccessFile.close();
      bufferT.get().clear();
      mdT.get().reset();
  
      return javax.xml.bind.DatatypeConverter.printHexBinary(mdbytes)
              .toLowerCase();
  
  }
  </code></pre>