title: How to deal with reading and processing huge text files without getting OutofMemoryError
body: <p>I wrote some straightforward code to read text files (>1g) and do some processing on Strings.</p>
  
  <p>However, I have to deal with Java heap space problems since I try to append Strings (using StringBuilder) that are getting to big on memory usage at some point. I know that I can increase my heap space with, e. g. '-Xmx1024', but I would like to work with only little memory usage here.How could I change my code below to manage my operations?</p>
  
  <p>I am still a Java novice and maybe I made some mistakes in my code which may seem obvious to you.</p>
  
  <p>Here's the code snippet:</p>
  
  <pre><code>    private void setInputData() {
  
      Pattern pat = Pattern.compile("regex");
      BufferedReader br = null;
      Matcher mat = null;
  
      try {
          File myFile = new File("myFile");
          FileReader fr = new FileReader(myFile);
  
          br = new BufferedReader(fr);
          String line = null;
          String appendThisString = null;
          String processThisString = null;
          StringBuilder stringBuilder = new StringBuilder();
  
          while ((line = br.readLine()) != null) {
  
              mat = pat.matcher(line);
  
              if (mat.find()) {
                  appendThisString = mat.group(1);
              }
  
              if (line.contains("|")) {
                  processThisString = line.replace(" ", "").replace("|", "\t");
                  stringBuilder.append(processThisString).append("\t").append(appendThisString);
                  stringBuilder.append("\n");
              }
          }
  //      doSomethingWithTheString(stringBuilder.toString());
      } catch (Exception ex) {
          ex.printStackTrace();
      } finally {
          try {
              if (br != null)br.close();
          } catch (IOException ex) {
              ex.printStackTrace();
          }
      }
  }
  </code></pre>
  
  <p>Here's the error message:</p>
  
  <pre>
  Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Arrays.java:2367)
      at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
      at java.lang.StringBuilder.append(StringBuilder.java:132)
      at Test.setInputData(Test.java:47)
      at Test.go(Test.java:18)
      at Test.main(Test.java:13)
  </pre>