title: Crawling hidden data using Scrapy
body: <p>I just started crawling. I'm trying to crawl question and answers from website <a href="http://www.indiabix.com/verbal-ability/spotting-errors/" rel="nofollow">http://www.indiabix.com/verbal-ability/spotting-errors/</a> by downloading content using Scrapy framework and Python 2.7. I noticed that if you view its source, you'll notice that answer for every question should be in the b tag but its not:</p>
  
  <pre><code>&lt;div class="div-spacer"&gt;
     &lt;p&gt;&lt;span class="ib-green"&gt;&lt;b&gt;Answer:&lt;/b&gt;&lt;/span&gt; Option &lt;b class="jq-hdnakqb"&gt;&lt;/b&gt;&lt;/p&gt; 
     &lt;p&gt;&lt;span class="ib-green"&gt;&lt;b&gt;Explanation:&lt;/b&gt;&lt;/span&gt;&lt;/p&gt; 
     &lt;p&gt; No answer description available for this question. &lt;b&gt;&lt;a href="discussion-399"&gt;Let us discuss&lt;/a&gt;&lt;/b&gt;. &lt;/p&gt;
  </code></pre>
  
  <p>If we inspect element on the webpage we can see the correct answer as text between the  tags : Answer: Option A or B etc. for each question but the HTML source code doesn't. </p>
  
  <p>To get the text within the b tag I've tried around 15 queries using xpath.
  Ive written the most probable 4-5 queries as comments in the code below.</p>
  
  <pre><code>import scrapy
  import urllib
  import json
  from errors1.items import Errors1Item
  
  class Errors1Spider(scrapy.Spider) :
      name = "errors1"
      start_urls =  ["http://www.indiabix.com/verbal-ability/spotting-errors/"]
  
      def parse(self, response) :
          i = 0
          y = 0
          j = json.loads(json.dumps(response.xpath('//td[contains(@id, "tdOption")]/text()').extract()))
          x = json.loads(json.dumps(response.xpath('//div[@class="div-spacer"]/p[3]/text()').extract()))
  
          #to get correct answer
          #response.xpath('//div[@class = "div-spacer"]/p/b/text()').extract()
          #response.xpath('//div[@class = "div-spacer"]/p[1]/b/text()').extract()
          #response.xpath('//div[@class = "div-spacer"]/p//text()').extract()
          #response.xpath('//b[@class = "jq-hdnakqb"]/text()').extract()
          #response.xpath('string(//div[@class = "div-spacer"]/p/b/text())').extract()
  
          while i&lt;len(j) and y&lt;len(x) :
              item = Errors1Item()
              item['optionA'] = j[i]
              i+=1 
              item['optionB'] = j[i]
              i+=1 
              item['optionC'] = j[i]
              i+=1 
              item['optionD'] = j[i]
              i+=1 
              item['explanation'] = x[y]
              y+=1
              yield item
  </code></pre>
  
  <p>Can someone please help me get the answer content from that webpage.
  Thanks</p>