title: Very fast webpage scraping (Python)
body: <p>So I'm trying to filter through a list of urls (potentially in the hundreds) and filter out every article who's body is less than X number of words (ARTICLE LENGTH). But when I run my application, it takes an unreasonable amount of time, so much so that my hosting service times out. I'm currently using Goose (<a href="https://github.com/grangier/python-goose" rel="nofollow">https://github.com/grangier/python-goose</a>) with the following filter function:</p>
  
  <pre><code>def is_news_and_data(url):
  """A function that returns a list of the form
      [True, title, meta_description]
      or
      [False]
  """
  result = []
  if url == None:
      return False
  try:
      article = g.extract(url=url)
      if len(article.cleaned_text.split()) &lt; ARTICLE_LENGTH:
          result.append(False)
      else:
          title = article.title
          meta_description = article.meta_description
          result.extend([True, title, meta_description])
  except:
      result.append(False)
  return result
  </code></pre>
  
  <p>In the context of the following. Dont mind the debug prints and messiness (tweepy is my twitter api wrapper):</p>
  
  <pre><code>def get_links(auth):
  """Returns a list of t.co links from a list of given tweets"""
  api = tweepy.API(auth)
  page_list = []
  tweets_list = []
  links_list = []
  news_list = []
  regex = re.compile('http://t.co/.[a-zA-Z0-9]*')
  
  for page in tweepy.Cursor(api.home_timeline, count=20).pages(1):
      page_list.append(page)
  for page in page_list:
      for status in page:
          tweet = status.text.encode('utf-8','ignore')
          tweets_list.append(tweet)
  
  for tweet in tweets_list:
      links = regex.findall(tweet)
      links_list.extend(links)
  #print 'The length of the links list is: ' + str(len(links_list))
  for link in links_list:
      news_and_data = is_news_and_data(link)
      if True in news_and_data:
          news_and_data.append(link)
          #[True, title, meta_description, link]
          news_list.append(news_and_data[1:])
  print 'The length of the news list is: ' + str(len(news_list))
  </code></pre>
  
  <p>Can anyone recommend a perhaps faster method?</p>