title: How to use scrapy to scrape SO question and store in MongoDB?
body: <p>This <a href="https://realpython.com/web-scraping-with-scrapy-and-mongodb/" rel="nofollow noreferrer">web scraping with Scrapy</a> is a little bit outdated link
  It seems that Selector XPath has been changed.</p>
  
  <p>When I copy it,I have</p>
  
  <pre><code>def parse(self, response):
      questions = Selector(response).xpath('//*[@id="question-header"]/h1/a')
  </code></pre>
  
  <p>But code from above link</p>
  
  <pre><code>class StackSpider(Spider):
      name = "stack"
      allowed_domains = ["stackoverflow.com"]
      start_urls = [
          "http://stackoverflow.com/questions?pagesize=50&amp;sort=newest",
      ]
  
      def parse(self, response):
          questions = Selector(response).xpath('//div[@class="summary"]/h3')
  
          for question in questions:
              item = StackItem()
              item['title'] = question.xpath(
                  'a[@class="question-hyperlink"]/text()').extract()[0]
              item['url'] = question.xpath(
                  'a[@class="question-hyperlink"]/@href').extract()[0]
              yield item
  </code></pre>
  
  <p>How to constructor generator with new Selector?</p>
  
  <p>This is</p>
  
  <p><a href="https://stackoverflow.com/questions/27624141/spring-data-transactional-not-rolling-back-with-sql-server-and-after-runtimeexc">Spring data @transactional not rolling back with SQL Server and after runtimeexception</a></p>
  
  <p>the SO question we are scraping as an example.</p>
  
  <p>Matthew Daniels suggestions</p>
  
  <pre><code>In [4]: response                                                                                                                                                                                            
  Out[4]: &lt;200 https://stackoverflow.com/questions/27624141/spring-data-transactional-not-rolling-back-with-sql-server-and-after-runtimeexc&gt;
  
  In [5]: response.css(".question-hyperlink").xpath("@href").extract_first()                                                                                                                                  
  Out[5]: '/questions/27624141/spring-data-transactional-not-rolling-back-with-sql-server-and-after-runtimeexc'
  
  In [6]: response.css(".summary h3")                                                                                                                                                                         
  Out[6]: []
  
  In [7]: response.css("#question-header &gt; h1 &gt; a")                                                                                                                                                           
  Out[7]: [&lt;Selector xpath="descendant-or-self::*[@id = 'question-header']/h1/a" data='&lt;a href="/questions/27624141/spring-data'&gt;]
  </code></pre>