title: Regex: Match a string up to a pattern (Which may not exist)
body: <p>I'm trying to parse an XML document using a regex tokenizer in Python (It's a finite set, so regex is just fine!), and I'm having trouble matching comments properly.</p>
  
  <p>The format of these comments is in the form <code>&lt;!--This is a comment--&gt;</code> where the comment itself can contain all sorts of non-alphanumeric characters (including '-')</p>
  
  <p>I want to match them in such a way that I break the comment up into the following tokens:</p>
  
  <p><code>&lt;!--</code></p>
  
  <p><code>This is a comment</code></p>
  
  <p><code>--&gt;</code></p>
  
  <p>The beginning marker is easy enough to get, and I successfully grab that with another regex, but the comment regex itself is getting too greedy and grabbing the <code>--</code> from the end marker. I want this regex to grab strings that are not necessarily enclosed in a comment as well, though, so it should also be able to take <code>&lt;Tag&gt;This is text&lt;/Tag&gt;</code> and correctly return <code>This is text</code>.</p>
  
  <p>This is the regex I am currently using for the text:</p>
  
  <p><code>[^&lt;&gt;]+(?!--&gt;)</code></p>
  
  <p>The end result ends up being <code>This is a comment--</code>, when I just want <code>This is a comment</code> so that my other regex can grab the <code>--&gt;</code>. This regex does work for normal tags, however, due to the existence of the '&lt;' on the end tag and properly returns <code>This is text</code> from my previous example.</p>
  
  <p>I know I must not be using negative lookahead properly. Any ideas on what I am doing wrong here? I've tried <code>[^&lt;&gt;]+(?=--&gt;)</code>, but then that doesn't match anything that's not a comment of this form (like the normal tags). I figured that the <code>(?!--&gt;)</code> would stop the matching when it sees that pattern, but it doesn't seem to work like that and just continues matching until it sees the ending '>'.</p>
  
  <p>Posting a segment of code for context:</p>
  
  <pre><code>xml_scanner = re.Scanner([
      (r"  ",             lambda scanner,token:("INDENT", token)),
      (r"&lt;[A-Za-z\d._]+(?!\/)&gt;",  lambda scanner,token:("BEGINTAG", token)),
      (r"&lt;\/[A-Za-z\d._]+(?!\/)&gt;",  lambda scanner,token:("ENDTAG", token)),
      (r"&lt;[A-Za-z\d._]+\/&gt;",      lambda scanner,token:("INLINETAG", token)),
      (r"&lt;!--",               lambda scanner,token:("BEGINCOMMENT", token)),
      (r"--&gt;",                lambda scanner,token:("ENDCOMMENT", token)),
      (r"[^&lt;&gt;]+(?!--&gt;)",         lambda scanner,token:("DATA", token)),
      (r"\r$", None),
  ])
  
  for line in database_file:
      results, remainder = xml_scanner.scan(line)
  </code></pre>
  
  <p>This is the only thing the script is doing at the moment.</p>