title: Unexpected StreamTokenizer behavior in Android
body: <p>I'm encountering this bizarre problem: the same code produces different results in Native Java than in Android. </p>
  
  <pre><code>InputStreamReader reader = new InputStreamReader(in, "UTF-8");
  BufferedReader m_reader = new BufferedReader(reader);
  StreamTokenizer m_tokenizer = new StreamTokenizer(m_reader);
  m_tokenizer.nextToken();
  System.out.println(m_tokenizer.toString());
  m_tokenizer.nextToken();
  System.out.println(m_tokenizer.toString());
  m_tokenizer.nextToken();
  System.out.println(m_tokenizer.toString());
  m_tokenizer.nextToken();
  System.out.println(m_tokenizer.toString());
  int c = m_reader.read();
  System.out.println(c);
  m_tokenizer.nextToken();
  System.out.println(m_tokenizer.toString());
  m_tokenizer.nextToken();
  System.out.println(m_tokenizer.toString());
  </code></pre>
  
  <p>Given the following Inputstream (read from a file)</p>
  
  <pre><code>(;FF[4]CA[UTF-8]
  </code></pre>
  
  <p>Native Java prints out </p>
  
  <pre><code>Token['('], line 1
  Token[';'], line 1
  Token[FF], line 1
  Token['['], line 1
  52
  Token[']'], line 1
  Token[CA], line 1
  </code></pre>
  
  <p>as expected. But in Android I got: </p>
  
  <pre><code>Token['('], line 1
  Token[';'], line 1
  Token[FF], line 1
  Token['['], line 1
  93
  Token[n=4.0], line 1
  Token[CA], line 1
  </code></pre>
  
  <p>Why does it behave differently in Android Java? In Android, somehow the character ']' is taken out from the stream before the tokenizer got there. I have read Java docs and Android docs and those classes seem to be identical. </p>
  
  <p>My API level is set to 7. And I've tried on both Android 2.1 Emulator and Android 4.0 Emulator getting the same result. I've also tried running it on a real device and I got the same result as well. </p>