title: Cannot Scrape IMDB after logging into personal profile using Scrapy?
body: <p>Following is my code. I am not able to scrape IMDB after I login successfully. The problem is that the after_login verifies that the Form request has worked but when I print the content of the page after login by making a new Request, it shows the main IMDB page and not the main IMDB page with user logged in. </p>
  
  <pre><code>"""
  Attributes:
      name (str): essential attribute which specifies the name of the spider
      start_urls (list): the urls that are to be scraped
  
  """
  name = 'IMDB_spider'
  start_urls = ['https://www.imdb.com/ap/signin?clientContext=131-8656718-8097200&amp;openid.pape.max_auth_age=0&amp;openid.'
              'return_to=https%3A%2F%2Fwww.imdb.com%2Fap-signin-handler&amp;openid.identity=http%3A%2F%2Fspecs.openid.ne'
              't%2Fauth%2F2.0%2Fidentifier_select&amp;openid.assoc_handle=imdb_us&amp;openid.mode=checkid_setup&amp;siteState=ey'
              'JvcGVuaWQuYXNzb2NfaGFuZGxlIjoiaW1kYl91cyIsInJlZGlyZWN0VG8iOiJodHRwczovL3d3dy5pbWRiLmNvbS8_cmVmXz1sb2d'
              'pbiJ9&amp;openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&amp;openid.ns=http'
              '%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&amp;&amp;tag=imdbtag_reg-20']
  
  def parse(self, response):
      """
      Scrapy's default method that handles all the downloaded response for
      each request made.
  
      Arguments:
          response (text): contains all data of the page and other helpful
          methods as well
  
      """
      return scrapy.FormRequest.from_response(
          response,
          formdata={'username': '*******', 'password': '****'},
          callback=self.after_login
      )
  
  def after_login(self, response):
      """
      Default callback method that is called to authenticate when logging in
      to website.
  
      Arguments:
          response (text): contains all data of the page and other helpful
          methods as well
  
      """
      if "There was a problem." in response.body:
          print('Login Failed')
          return
      print('Login Success')
      return scrapy.Request(url="http://www.imdb.com",
                            callback=self.parse_imdb_page)
  
  def parse_imdb_page(self, response):
      print response.body
  </code></pre>
  
  <p>Please help</p>