title: Scrapy not scraping https?
body: <p>New to scrapy so I may just be doing stuff wrong. It seems however that scrapy won't scrape any https sites I feed into it.</p>
  <pre><code>class SeleniumSpider(CrawlSpider):
      name = &quot;SeleniumSpider&quot;
      start_urls = [&quot;https://www.facebook.com&quot;]
  
      rules = (
          Rule(SgmlLinkExtractor(allow=('\.html', )), callback='parse_page',follow=True),
      )
  
      def __init__(self):
          CrawlSpider.__init__(self)
          
      def __del__(self):
          self.driver.stop()
          print self.verificationErrors
          CrawlSpider.__del__(self)
  
      def parse_page(self, response):
          hxs = HtmlXPathSelector(response)
          hxs.select('//div').extract()
  </code></pre>
  <p>Output:</p>
  <pre><code>2014-05-30 11:22:01-0400 [scrapy] INFO: Scrapy 0.22.2 started (bot: scrapybot)
  2014-05-30 11:22:01-0400 [scrapy] INFO: Optional features available: ssl, http11
  2014-05-30 11:22:01-0400 [scrapy] INFO: Overridden settings: {'DEFAULT_ITEM_CLASS': 'dirbot.items.Website', 'NEWSPIDER_MODULE': 'dirbot.spiders', 'SPIDER_MODULES': ['dirbot.spiders']}    
  2014-05-30 11:22:01-0400 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState  
  2014-05-30 11:22:01-0400 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats  
  2014-05-30 11:22:01-0400 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
  2014-05-30 11:22:01-0400 [scrapy] INFO: Enabled item pipelines: FilterWordsPipeline
  2014-05-30 11:22:01-0400 [SeleniumSpider] INFO: Spider opened
  2014-05-30 11:22:01-0400 [SeleniumSpider] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2014-05-30 11:22:01-0400 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
  2014-05-30 11:22:01-0400 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
  2014-05-30 11:22:01-0400 [SeleniumSpider] DEBUG: Crawled (200) &lt;GET https://www.facebook.com&gt; (referer: None)
  2014-05-30 11:22:01-0400 [SeleniumSpider] INFO: Closing spider (finished)
  2014-05-30 11:22:01-0400 [SeleniumSpider] INFO: Dumping Scrapy stats:
  </code></pre>
  <p>Any suggestions? Crawler works fine on <a href="http://www.amazon.com" rel="nofollow noreferrer">http://www.amazon.com</a> and others</p>