title: Java Tokenizer, separating strings
body: <p>I don't know why I am struggling so badly with this but any help would be much appreciated. </p>
  
  <p>I am creating my own tokenizer that takes in a file with a list of commands, delimiters and values. It then outputs each "token" along with what type it is.</p>
  
  <p>INPUT: <code>AND 3, 4, 5 ; some comments</code></p>
  
  <p>I need to output: </p>
  
  <pre><code>AND --- command
  3 --- value
  , --- delimiter
  4 --- value
  , --- delimiter
  5 --- value
  </code></pre>
  
  <p>I have it working right now to where I am outputting:</p>
  
  <pre><code>AND 3, 4, 5 --- delimiter
  </code></pre>
  
  <p>but I need to break it down further.</p>
  
  <p>Here is where I am at currently:</p>
  
  <pre><code>ArrayList&lt;Token&gt; tokenize(String[] input) {
      ArrayList&lt;Token&gt; tokens = new ArrayList&lt;Token&gt;();
      for (String str : input) {
          Token token = new Token(str.trim());
          //Check if int
          try{
              Integer.parseInt(str);
              token.type = "number";
          } catch(NumberFormatException e) {
  
          }
          if (token.type == null) {
              if (commands.contains(str))
                  token.type = "command";
               else if (str.contains(",")) {
                  token.type = "delimiter";
              } else if (destValues.contains(str))
                  token.type = "destination";
              else
                  token.type = "unknown";
          }
  
          if(! token.type.equals("unknown"))
              tokens.add(token);
      }
      return tokens;
  }
  </code></pre>
  
  <p>Only real constraints I have with this assignment is not being able to use StringTokenizer and regex.</p>