title: crawlspider not crawling using urls in text files
body: <p>Problem Statement:</p>
  
  <p>I have a list of forum urls in a file names myurls.csv in each line as follows:</p>
  
  <pre><code>https://www.drupal.org/user/3178461/track
  https://www.drupal.org/user/511008/track
  </code></pre>
  
  <p>I wrote a CrawlSpider code to crawl the forum posts as follows:</p>
  
  <pre><code>class fileuserurl(CrawlSpider):
      name = "fileuserurl"
      allowed_domains = []
      start_urls = []
  
      rules = (
      Rule(SgmlLinkExtractor(allow=('/user/\d/track'),
      restrict_xpaths = ('//li[@class="pager-next"]',),
      canonicalize=False ),callback='parse_page',follow=True)
      )
  
      def __init__(self):
      f = open('./myurls.txt','r').readlines()
      self.allowed_domains = ['www.drupal.org']
      self.start_urls = [l.strip() for l in f]
      super(fileuserurl,self).__init__()
  
      def parse_page(self, response):
      print '*********** START PARSE_PAGE METHOD**************'
      # print response.url
      items = response.xpath("//tbody/tr")
      myposts=[]
      for temp in items:
      item = TopicPosts()
      item['topic'] = temp.xpath(".//td[2]/a/text()").extract()
      relative_url = temp.xpath(".//td[2]/a/@href").extract()[0]
      item['topiclink'] = 'https://www.drupal.org'+relative_url
      item['author'] = temp.xpath(".//td[3]/a/text()").extract()
      try:
      item['replies'] = str(temp.xpath(".//td[4]/text()").extract()[0]).strip('\n')
      except:
      continue
      myposts.append(item)
      return myposts
  </code></pre>
  
  <p>Question:</p>
  
  <p>It only gives me the first page output of the urls mentioned in my text file. I want to go to every link of the pages as defined by next in the front page.</p>