title: Returning java regex (words, spaces, special characters, double quotes)
body: <p>I am trying to use java regex to tokenize any language source file. What I want the list to return is:</p>
  
  <ul>
  <li>words (<code>[a-z_A-Z0-9]</code>)</li>
  <li>spaces</li>
  <li>any of <code>[()*.,+-/=&amp;:]</code> as a single character</li>
  <li>and quoted items left in quotes.</li>
  </ul>
  
  <p>Here is the code I have so far:</p>
  
  <pre><code>Pattern pattern = Pattern.compile("[\"(\\w)\"]+|[\\s\\(\\)\\*\\+\\.,-/=&amp;:]");
  
  Matcher matcher = pattern.matcher(str);
  List&lt;String&gt; matchlist = new ArrayList&lt;String&gt;();
  
  while(matcher.find()) {
      matchlist.add(matcher.group(0));
  }
  </code></pre>
  
  <p>For example, </p>
  
  <pre><code>"I" am_the 2nd "best".
  </code></pre>
  
  <p>returns: list, size 8</p>
  
  <pre><code>("I", ,am_the, ,2nd, ,"best", .)
  </code></pre>
  
  <p>which is what I want. However, if the whole sentence is quoted, except for the period:</p>
  
  <pre><code>"I am_the 2nd best".
  </code></pre>
  
  <p>returns: list, size 8</p>
  
  <pre><code>("I, ,am_the, ,2nd, ,best", .)
  </code></pre>
  
  <p>and I want it to be able to return: list, size 2</p>
  
  <pre><code>("I am_the 2nd best", .)
  </code></pre>
  
  <p>If that makes sense. I believe it works for everything I want it to except for returning string literals (which I want to keep the quotes). What is it that I am missing from the pattern that will allow me to achieve this? </p>
  
  <p>And by all means, if there is an easier pattern to use that I do not see, please help me out. The pattern shown above was the compilation of many trial/error. Thank you very much in advance for any help.</p>