title: Python, Matching uneven length scraped lists
body: <p>Please prepare for a long read.  I am at a standstill and don't know where to look at for an answer / what else to try.  Needless to say I am kinda new to programming.  Been hacking away at this project for the past couple of weeks.</p>
  
  <p><strong>PROBLEM</strong></p>
  
  <p>I got this table, 25 lines, 2 columns.  Each line is structured like:</p>
  
  <p><b> Needed event </b>  </p>
  
  <pre><code>&lt;td align=center&gt;19/11/11&lt;br&gt;12:01:21 AM&lt;/td&gt;
  &lt;td align=center&gt;&lt;font color=#006633&gt;&lt;a href=profiles.php?XID=1&gt;&lt;font color=#006633&gt;player1&lt;/font&gt;&lt;/a&gt; hospitalized &lt;a href=profiles.php?XID=2&gt;&lt;font color=#006633&gt;player2&lt;/font&gt;&lt;/a&gt;&lt;/font&gt;&lt;/td&gt;
  </code></pre>
  
  <p><b> NOT Needed event 
  CASE A</b></p>
  
  <pre><code>&lt;td align="center"&gt;19/11/11&lt;br /&gt;12:58:03 AM&lt;/td&gt;
  &lt;td align="center"&gt;&lt;font color="#AA0000"&gt;Someone hospitalized &lt;a href=profiles.php?XID=1&gt;&lt;font color="#AA0000"&gt;player1&lt;/font&gt;&lt;/a&gt;&lt;/font&gt;&lt;/td&gt;
  </code></pre>
  
  <p><b> NOT Needed event 
  CASE B</b></p>
  
  <pre><code>&lt;td align="center"&gt;19/11/11&lt;br /&gt;12:58:03 AM&lt;/td&gt;
  &lt;td align=center&gt;&lt;font color=#006633&gt;&lt;a href=profiles.php?XID=3&gt;&lt;font color=#006633&gt;player3&lt;/font&gt;&lt;/a&gt; attacked &lt;a href=profiles.php?XID=1&gt;&lt;font color=#006633&gt;player1&lt;/font&gt;&lt;/a&gt; and lost &lt;/font&gt;&lt;/td&gt;
  </code></pre>
  
  <p>I have used regex to scrape the needed data.  My problem is that the 2 lists are not evently matched.  Date and time don't always match to the exact event.</p>
  
  <hr>
  
  <p><strong>1st ATTEMPT at solving problem</strong></p>
  
  <pre><code>import mechanize  
  import re
  
  htmlA1 = br.response().read()
  
  patAttackDate = re.compile('&lt;td align=center&gt;(\d+/\d+/\d+)&lt;br&gt;(\d+:\d+:\d+ \w+)')
  patAttackName = re.compile('&lt;font color=#006633&gt;(\w+)&lt;/font&gt;&lt;/a&gt; hospitalized ')
  searchAttackDate = re.findall(patAttackDate, htmlA1)
  searchAttackName = re.findall(patAttackName, htmlA1)
  
  pairs = zip(searchAttackDate, searchAttackName)
  
  for i in pairs:
  print (i)
  </code></pre>
  
  <p>But that gets me a <code>wrong time - correct event</code> type of list.</p>
  
  <p>for example:</p>
  
  <pre><code>(('19/11/11', '9:47:51 PM'), 'user1') &lt;- mismatch 
  (('19/11/11', '8:21:18 PM'), 'user1') &lt;- mismatch
  (('19/11/11', '7:33:00 PM'), 'user1') &lt;- As a consequence of the below, the rest upwards are mismatched 
  (('19/11/11', '7:32:38 PM'), 'user2') &lt;- NOT a match, case B
  (('19/11/11', '7:32:22 PM'), 'user2') &lt;- match ok
  (('19/11/11', '7:26:53 PM'), 'user2') &lt;- match ok
  (('19/11/11', '7:25:24 PM'), 'user3') &lt;- match ok
  (('19/11/11', '7:24:22 PM'), 'user3') &lt;- match ok
  (('19/11/11', '7:23:25 PM'), 'user3') &lt;- match ok
  </code></pre>
  
  <hr>
  
  <p><strong>2nd ATTEMPT at solving problem</strong></p>
  
  <p>So thought to strip the <code>newline</code> from the whole page and scrape the table, but:</p>
  
  <pre><code>import mechanize
  import re
  from BeautifulSoup import BeautifulSoup
  
  htmlA1 = br.response().read()
  
  stripped = htmlA1.replace("&gt;\n&lt;","&gt;&lt;") #Removed all '\n' from code
  
  soup = BeautifulSoup(stripped)
  
  table = soup.find('table', width='90%')
  table2 = table.findNext('table', width='90%')
  table3 = table2.findNext('table', width='90%') #this is the table I need to work with
  
  patAttackDate = re.compile('&lt;td align="center"&gt;(\d+/\d+/\d+)&lt;br /&gt;(\d+:\d+:\d+ \w+)')
  searchAttackDate = re.findall(patAttackDate, table3)
  print searchAttackDate
  </code></pre>
  
  <p>this gives me an error: </p>
  
  <pre><code>return _compile(pattern, flags).findall(string)
  TypeError: expected string or buffer
  </code></pre>
  
  <p>What am I missing?</p>
  
  <p>Bonus question:
  Is there any way to account for XID being a dynamic variable but bypass it when using regex / beautifulsoup (or other scraping method)?  As the project 'grows' I might need to include the XID portion of code but don't want to match to it. (not sure if this is clear)</p>
  
  <p>Thank you for your time</p>
  
  <hr>
  
  <p><b>EDIT 1</b>: Added list example<br>
  <strong>EDIT 2</strong>: Made code separation more visible<br>
  <strong>EDIT 3</strong>: Added sample code for a given solution that doesn't seem to work</p>
  
  <pre><code>Test = '''&lt;table&gt;&lt;tr&gt;&lt;td&gt;date&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;'''
  soupTest = BeautifulSoup(Test)
  test2 = soupTest.find('table')
  patTest = re.compile('&lt;td&gt;(.*)&lt;/td&gt;')
  searchTest = patTest.findall(test2.getText())
  print test2 # gives: &lt;table&gt;&lt;tr&gt;&lt;td&gt;date&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt; 
  print type(test2) # gives: &lt;class 'BeautifulSoup.Tag'&gt;
  print searchTest #gives: []
  </code></pre>
  
  <p><strong>EDIT 4 - Solution</strong></p>
  
  <pre><code>import re
  import mechanize
  from BeautifulSoup import BeautifulSoup
  
  htmlA1 = br.response().read()
  stripped = htmlA1.replace("&gt;\n&lt;","&gt;&lt;") #stripped '\n' from html
  soup = BeautifulSoup(stripped)
  
  table = soup.find('table', width='90%')
  table2 = table.findNext('table', width='90%')
  table3 = table2.findNext('table', width='90%') #table I need to work with
  
  print type(table3) # gives &lt;class 'BeautifulSoup.Tag'&gt;
  strTable3 = str(table3) #convert table3 to string type so i can regex it
  
  patFinal = re.compile(('(\d+/\d+/\d+)&lt;br /&gt;(\d+:\d+:\d+ \w+)&lt;/td&gt;&lt;td align="center"&gt;'
                        '&lt;font color="#006633"&gt;&lt;a href="profiles.php\?XID=(\d+)"&gt;'
                        '&lt;font color="#006633"&gt;(\w+)&lt;/font&gt;&lt;/a&gt; hospitalized &lt;a'), re.IGNORECASE)
  searchFinal = re.findall(patFinal, strTable3)
  
  for i in searchFinal:
      print (i)
  </code></pre>
  
  <hr>
  
  <p><strong>Sample output</strong></p>
  
  <pre><code>('19/11/11', '1:08:07 AM', 'ID_user1', 'user1')
  ('19/11/11', '1:06:55 AM', 'ID_user1', 'user1')
  ('19/11/11', '1:05:46 AM', 'ID_user1', 'user1')
  ('19/11/11', '1:04:33 AM', 'ID_user1', 'user1')
  ('19/11/11', '1:03:32 AM', 'ID_user1', 'user1')
  ('19/11/11', '1:02:37 AM', 'ID_user1', 'user1')
  ('19/11/11', '1:00:43 AM', 'ID_user1', 'user1')
  ('19/11/11', '12:55:35 AM', 'ID_user2', 'user2')
  </code></pre>
  
  <p><strong>EDIT 5 - A much simpler solution (on 1st attempt - without Beautifulsoup)</strong></p>
  
  <pre><code>import re
  
  reAttack = (r'&lt;td\s+align=center&gt;(\d+/\d+/\d+)&lt;br&gt;(\d+:\d+:\d+\s+\w+)&lt;/td&gt;\s*'
              '&lt;td.*?' #accounts for the '\n'
              '&lt;font\s+color=#006633&gt;(\w+)&lt;/font&gt;&lt;/a&gt;\s+hospitalized\s+')
  
  for m in re.finditer(reAttack, htmlA1):
      print 'date: %s; time: %s; player: %s' % (m.group(1), m.group(2), m.group(3))
  </code></pre>
  
  <p><strong>Sample Output</strong></p>
  
  <pre><code>date: 19/11/11; time: 1:08:07 AM; player: user1
  date: 19/11/11; time: 1:06:55 AM; player: user1
  date: 19/11/11; time: 1:05:46 AM; player: user1
  date: 19/11/11; time: 1:04:33 AM; player: user1
  date: 19/11/11; time: 1:03:32 AM; player: user1
  date: 19/11/11; time: 1:02:37 AM; player: user1
  date: 19/11/11; time: 1:00:43 AM; player: user1
  date: 19/11/11; time: 12:55:35 AM; player: user2
  </code></pre>