Simple filter to remove obviously bad paragraphs (bad text extraction).

  Note this needs to run very quickly as it is applied to every paragraph
  in the corpus, so nothing fancy! This whole method should be linear
  expected time in len(p).

  Args:
    p: string, paragraph

  Returns:
    True if we should remove the paragraph.

def filter_paragraph(p):
  """Simple filter to remove obviously bad paragraphs (bad text extraction).

  Note this needs to run very quickly as it is applied to every paragraph
  in the corpus, so nothing fancy! This whole method should be linear
  expected time in len(p).

  Args:
    p: string, paragraph

  Returns:
    True if we should remove the paragraph.
  """
  # Expect a minimum number of words.
  tokens = p.split()
  if len(tokens) < 6:
    return True

  # Require some letters.
  if not re.search(_SOME_ALPHA_RE, p):
    return True

  # Keep this one at the end, probably the most complicated logic.
  # We try to detect sentences, which should have a minimum of 3 tokens
  # with only alphabetic characters.
  last = 0
  found_sentence = False
  num_alpha = 0
  for i, x in enumerate(tokens):
    if x == '.':
      if i - last > 3 and num_alpha >= 3:
        found_sentence = True
        break
      last = i
      num_alpha = 0
    if re.match(_ONLY_ALPHA_RE, x):
      num_alpha += 1
  if not found_sentence:
    return True

  return False