title: Tokenizer for math expression
body: <p>Using the <code>StreamTokenizer</code> I am writing a lexer that will tokenize a math expression.</p>
  
  <p>As an input, I give the expression <code>(1+π)²(1−π)²+(5.3−-2)/6</code>. 
  I expect it to be tokenized as <br>
  <code>( 1 + π ) ² ( 1 - π ) ² + ( 5.3 − - 2 ) / 6</code> <br> 
  but I get <code>( 1 +π ) ² ( 1 -π ) ² + ( 5.3 −-2 ) / 6</code>.</p>
  
  <p>I know I need to insert the multiplication operators in certain places of the output, will do it later.</p>
  
  <pre><code>/* s: The inputted expression */
  public static String tokenize(String s)[] throws IOException
  {
      StreamTokenizer tokenizer = new StreamTokenizer(new StringReader(s));
      tokenizer.parseNumbers();
      tokenizer.wordChars('a', 'z');
      tokenizer.wordChars('A', 'Z');
      tokenizer.wordChars('A', 'Z');
      tokenizer.wordChars(SQUARED, SQUARED); // the superscript 2
      tokenizer.wordChars(PI, PI);
      tokenizer.wordChars(SUB.charAt(0), SUB.charAt(0)); // subtract (takeaway)
      tokenizer.wordChars(NEG.charAt(0), NEG.charAt(0)); // negate
      tokenizer.wordChars('/', '/');
      tokenizer.wordChars('*', '*');
      tokenizer.wordChars('+', '+');
      tokenizer.ordinaryChar(',');
      tokenizer.ordinaryChar('/');    // do not consider / as comment start
  
      ArrayList&lt;String&gt; tokBuf = new ArrayList&lt;&gt;();
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
          switch (tokenizer.ttype) {
              case StreamTokenizer.TT_NUMBER:
                  tokBuf.add(String.valueOf(tokenizer.nval));
                  break;
              case StreamTokenizer.TT_WORD:
                  tokBuf.add(tokenizer.sval);
                  break;
              default:
                  tokBuf.add(String.valueOf((char) tokenizer.ttype));
          }
      }
      String ret[] = new String[tokBuf.size()];
      ret = tokBuf.toArray(ret);
  
      return ret;
  }
  </code></pre>