title: How to tokenize an input file in java
body: <p>i'm doing tokenizing a text file in java. I want to read an input file, tokenize it and write a certain character that has been tokenized into an output file. This is what i've done so far:</p>
  
  <pre><code>package org.apache.lucene.analysis;
  
  import java.io.BufferedReader;
  import java.io.File;
  import java.io.FileReader;
  import java.io.IOException;
  import java.io.InputStreamReader;
  import java.io.StreamTokenizer;
  
  class StringProcessing {
      // Create BufferedReader class instance
      public static void main(String[] args) throws IOException {
          InputStreamReader input = new InputStreamReader(System.in);
          BufferedReader keyboardInput = new BufferedReader(input);
          System.out.print("Please enter a java file name: ");
          String filename = keyboardInput.readLine();
          if (!filename.endsWith(".DAT")) {
              System.out.println("This is not a DAT file.");
              System.exit(0);
          }
          File File = new File(filename);
          if (File.exists()) {
              FileReader file = new FileReader(filename);
              StreamTokenizer streamTokenizer = new StreamTokenizer(file);
              int i = 0;
              int numberOfTokensGenerated = 0;
              while (i != StreamTokenizer.TT_EOF) {
                  i = streamTokenizer.nextToken();
                  numberOfTokensGenerated++;
              }
              // Output number of characters in the line
              System.out.println("Number of tokens = " + numberOfTokensGenerated);
              // Output tokens
              for (int counter = 0; counter &lt; numberOfTokensGenerated; counter++) {
                  char character = file.toString().charAt(counter);
                  if (character == ' ') { System.out.println(); } else { System.out.print(character); }
              }
          } else {
              System.out.println("File does not exist!");
              System.exit(0);
          }
  
          System.out.println("\n");
      }//end main
  }//end class
  </code></pre>
  
  <p>When i run this code, this is what i get:</p>
  
  <p><code>Please enter a java file name: D://eclipse-java-helios-SR1-win32/LexractData.DAT
  Number of tokens = 129
  java.io.FileReader@19821fException in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: 25
      at java.lang.String.charAt(Unknown Source)
      at org.apache.lucene.analysis.StringProcessing.main(StringProcessing.java:40)</code></p>
  
  <p>The input file will look like this:  </p>
  
  <pre><code>-K1 Account 
  --Op1 withdraw
  ---Param1 an
  ----Type Int
  ---Param2 amount
  ----Type Int
  --Op2 deposit
  ---Param1 an
  ----Type Int
  ---Param2 Amount
  ----Type Int
  --CA1 acNo
  ---Type Int
  -K2 CheckAccount 
  --SC Account
  --CA1 credit_limit
  ---Type Int
  -K3 Customer
  --CA1 name
  ---Type String
  -K4 Transaction
  --CA1 date
  ---Type Date
  --CA2 time
  ---Type Time
  -K5 CheckBook
  -K6 Check
  -K7 BalanceAccount
  --SC Account
  </code></pre>
  
  <p>I just want to read the string which are starts with <code>-K1</code>, <code>-K2</code>, <code>-K3</code>, and so on... can anyone help me?</p>