title: Ruby regular expression help using match to extract pieces of html doc
body: <p>I have an HTML document of this format:</p>
  
  <pre><code>&lt;tr&gt;&lt;td colspan="4"&gt;&lt;span class="fullName"&gt;Bill Gussio&lt;/span&gt;&lt;/td&gt;&lt;/tr&gt;
      &lt;tr&gt;
          &lt;td class="sectionHeader"&gt;Contact&lt;/td&gt;
          &lt;td class="sectionHeader"&gt;Phone&lt;/td&gt;
          &lt;td class="sectionHeader"&gt;Home&lt;/td&gt;
          &lt;td class="sectionHeader"&gt;Work&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr valign="top"&gt;
          &lt;td class="sectionContent"&gt;&lt;span&gt;Screen Name:&lt;/span&gt; &lt;span&gt;bhjiggy&lt;/span&gt;&lt;br&gt;&lt;span&gt;Email 1:&lt;/span&gt; &lt;span&gt;wmgussio@erols.com&lt;/span&gt;&lt;/td&gt;
          &lt;td class="sectionContent"&gt;&lt;span&gt;Mobile: &lt;/span&gt;&lt;span&gt;2404173223&lt;/span&gt;&lt;/td&gt;
          &lt;td class="sectionContent"&gt;&lt;span&gt;NY&lt;/span&gt;&lt;br&gt;&lt;span&gt;New York&lt;/span&gt;&lt;br&gt;&lt;span&gt;78642&lt;/span&gt;&lt;/td&gt;
          &lt;td class="sectionContent"&gt;&lt;span&gt;MD&lt;/span&gt;&lt;br&gt;&lt;span&gt;Owings Mills&lt;/span&gt;&lt;br&gt;&lt;span&gt;21093&lt;/span&gt;&lt;/td&gt;
      &lt;/tr&gt;
  
      &lt;tr&gt;&lt;td colspan="4"&gt;&lt;hr class="contactSeparator"&gt;&lt;/td&gt;&lt;/tr&gt;
  
      &lt;tr&gt;&lt;td colspan="4"&gt;&lt;span class="fullName"&gt;Eddie Osefo&lt;/span&gt;&lt;/td&gt;&lt;/tr&gt;
      &lt;tr&gt;
          &lt;td class="sectionHeader"&gt;Contact&lt;/td&gt;
          &lt;td class="sectionHeader"&gt;Phone&lt;/td&gt;
          &lt;td class="sectionHeader"&gt;Home&lt;/td&gt;
          &lt;td class="sectionHeader"&gt;Work&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr valign="top"&gt;
          &lt;td class="sectionContent"&gt;&lt;span&gt;Screen Name:&lt;/span&gt; &lt;span&gt;eddieOS&lt;/span&gt;&lt;br&gt;&lt;span&gt;Email 1:&lt;/span&gt; &lt;span&gt;osefo@wam.umd.edu&lt;/span&gt;&lt;/td&gt;
          &lt;td class="sectionContent"&gt;&lt;/td&gt;
          &lt;td class="sectionContent"&gt;&lt;span&gt;&lt;/span&gt;&lt;/td&gt;
          &lt;td class="sectionContent"&gt;&lt;span&gt;&lt;/span&gt;&lt;/td&gt;
      &lt;/tr&gt;
  
      &lt;tr&gt;&lt;td colspan="4"&gt;&lt;hr class="contactSeparator"&gt;&lt;/td&gt;&lt;/tr&gt;
  </code></pre>
  
  <p>So it alternates - chunk of contact info and then a "contact separator".  I want to grab the contact info so my first obstacle is to grab the chunks in between the contact separator.  I have already figured out the regular expression using rubular.  It is:</p>
  
  <pre><code>/&lt;tr&gt;&lt;td colspan="4"&gt;&lt;span class="fullName"&gt;((.|\s)*?)&lt;hr class="contactSeparator"&gt;/
  </code></pre>
  
  <p>You can check on rubular to verify that this isolates chunks.  </p>
  
  <p>However my big issue is that I am having trouble with the ruby code.  I use the built in match function and make prints, but do not get the results I expect.  Here is the code:</p>
  
  <pre><code>page = agent.get uri.to_s    
  chunks = page.body.match(/&lt;tr&gt;&lt;td colspan="4"&gt;&lt;span class="fullName"&gt;((.|\s)*?)&lt;hr class="contactSeparator"&gt;/).captures
  
  chunks.each do |chunk|
     puts "new chunk: " + chunk.inspect
  end
  </code></pre>
  
  <p>Note that page.body is just the body of the html document grabbed by Mechanize.  The html document is much larger but has this format.  So, the unexpected output is below:</p>
  
  <pre><code>new chunk: "Bill Gussio&lt;/span&gt;&lt;/td&gt;&lt;/tr&gt;\r\n\t&lt;tr&gt;\r\n\t\t&lt;td class=\"sectionHeader\"&gt;Contact&lt;/td&gt;\r\n\t\t&lt;td class=\"sectionHeader\"&gt;Phone&lt;/td&gt;\r\n\t\t&lt;td class=\"sectionHeader\"&gt;Home&lt;/td&gt;\r\n\t\t&lt;td class=\"sectionHeader\"&gt;Work&lt;/td&gt;\r\n\t&lt;/tr&gt;\r\n\t&lt;tr valign=\"top\"&gt;\r\n\t\t&lt;td class=\"sectionContent\"&gt;&lt;span&gt;Screen Name:&lt;/span&gt; &lt;span&gt;bhjiggy&lt;/span&gt;&lt;br&gt;&lt;span&gt;Email 1:&lt;/span&gt; &lt;span&gt;wmgussio@erols.com&lt;/span&gt;&lt;/td&gt;\r\n\t\t&lt;td class=\"sectionContent\"&gt;&lt;span&gt;Mobile: &lt;/span&gt;&lt;span&gt;2404173223&lt;/span&gt;&lt;/td&gt;\r\n\t\t&lt;td class=\"sectionContent\"&gt;&lt;span&gt;NY&lt;/span&gt;&lt;br&gt;&lt;span&gt;New York&lt;/span&gt;&lt;br&gt;&lt;span&gt;78642&lt;/span&gt;&lt;/td&gt;\r\n\t\t&lt;td class=\"sectionContent\"&gt;&lt;span&gt;MD&lt;/span&gt;&lt;br&gt;&lt;span&gt;Owings Mills&lt;/span&gt;&lt;br&gt;&lt;span&gt;21093&lt;/span&gt;&lt;/td&gt;\r\n\t&lt;/tr&gt;\r\n\t\r\n\t&lt;tr&gt;&lt;td colspan=\"4\"&gt;"
  new chunk: "&gt;"
  </code></pre>
  
  <p>There are 2 surprises here for me:</p>
  
  <p>1) There are not 2 matches that contain the chunks of contact info, even though on rubular I have verified that these chunks should be extracted.</p>
  
  <p>2) All of the \r\n\t (line feeds, tabs, etc.) are showing up in the matches.</p>
  
  <p>Can anyone see the issue here?</p>
  
  <p>Alternatively, if anyone knows of a good free AOL contacts importer, that would be great.  I have been using blackbook but it keeps failing for me on AOL and I am attempting to fix it.  Unfortunately, AOL has no contacts API yet.</p>
  
  <p>Thank you!</p>