title: How can I split a string into tokens?
body: <p>If I have a string</p>
  
  <pre><code>'x+13.5*10x-4e1'
  </code></pre>
  
  <p>how can I split it into the following list of tokens?</p>
  
  <pre><code>['x', '+', '13', '.', '5', '*', '10', 'x', '-', '4', 'e', '1']
  </code></pre>
  
  <p>Currently I'm using the shlex module:</p>
  
  <pre><code>str = 'x+13.5*10x-4e1'
  lexer = shlex.shlex(str)
  tokenList = []
  for token in lexer:
      tokenList.append(str(token))
  return tokenList
  </code></pre>
  
  <p>But this returns:</p>
  
  <pre><code>['x', '+', '13', '.', '5', '*', '10x', '-', '4e1']
  </code></pre>
  
  <p>So I'm trying to split the letters from the numbers. I'm considering taking the strings that contain both letters and numbers then somehow splitting them, but not sure about how to do this or how to add them all back into the list with the others afterwards. It's important that the tokens stay in order, and I can't have nested lists.</p>
  
  <p>In an ideal world, e and E would not be recognised as letters in the same way, so</p>
  
  <pre><code>'-4e1'
  </code></pre>
  
  <p>would become</p>
  
  <pre><code>['-', '4e1']
  </code></pre>
  
  <p>but</p>
  
  <pre><code>'-4x1'
  </code></pre>
  
  <p>would become</p>
  
  <pre><code>['-', '4', 'x', '1']
  </code></pre>
  
  <p>Can anybody help?</p>