title: How to scrape images from a aspx page?
body: <p>i am trying to <strong>scrape</strong> images from a aspx page i have this code that scrapes images from normal webpage but can't scrape aspx page cause i need to send http post requests to the aspx page i can't figure out how to do that even after reading few threads this is the original code</p>
  <pre><code>from bs4 import BeautifulSoup as bs
  import urlparse
  import urllib2
  from urllib import urlretrieve
  import os
  import sys
  import subprocess
  import re
  
  
  def thefunc(url, out_folder):
  
      c = False
  </code></pre>
  <p><strong>i have already defined headers for aspx page and a if statement that distinguished between normal page and aspx page</strong></p>
  <pre><code>    select =  raw_input('Is this a .net  aspx page ? y/n : ')
      if select.lower().startswith('y'):
          usin = raw_input('Specify origin of .net page : ')
          usaspx = raw_input('Specify aspx page url : ')
  </code></pre>
  <p><strong>the header for aspx page</strong></p>
  <pre><code>        headdic = {
              'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
              'Origin': usin,
              'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.17 (KHTML, like Gecko)  Chrome/24.0.1312.57 Safari/537.17',
              'Content-Type': 'application/x-www-form-urlencoded',
              'Referer': usaspx,
              'Accept-Encoding': 'gzip,deflate,sdch',
              'Accept-Language': 'en-US,en;q=0.8',
              'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
          }
          c = True
  
      if c:
          req = urllib2.Request(url, headers=headic)
      else:
          req = urllib2.Request(url, headers={'User-Agent' : &quot;Magic Browser&quot;})
      resp = urllib2.urlopen(req)
      
      soup = bs(resp, 'lxml')
      
      parsed = list(urlparse.urlparse(url))
  
      print '\n',len(soup.findAll('img')), 'images are about to be downloaded'
  
      for image in soup.findAll(&quot;img&quot;):
          
          print &quot;Image: %(src)s&quot; % image
          
          filename = image[&quot;src&quot;].split(&quot;/&quot;)[-1]
          
          parsed[2] = image[&quot;src&quot;]
          
          outpath = os.path.join(out_folder, filename)
  
          try:
          
              if image[&quot;src&quot;].lower().startswith(&quot;http&quot;):
                  urlretrieve(image[&quot;src&quot;], outpath)
              else:
                  urlretrieve(urlparse.urlunparse(parsed), outpath)
          except:
              print 'OOPS missed one for some reason !!'
              pass
  
  
  try:
      put =  raw_input('Please enter the page url : ')
      reg1 = re.compile('^http*',re.IGNORECASE)
      reg1.match(put)
  except:
      print('Type the url carefully !!')
      sys.exit()
  fol = raw_input('Enter the foldername to save the images : ')
  if os.path.isdir(fol):
      thefunc(put, fol)
  else:
      subprocess.call('mkdir', fol)
      thefunc(put, fol)
  </code></pre>
  <p><em><strong>i have made few modifications for aspx detection and creating the header for the aspx page but how to modify next i am stuck here</strong></em></p>
  <p><code>***here is the aspx page link***</code> <a href="http://www.foxrun.com.au/Products/Cylinders_with_Gadgets.aspx" rel="nofollow noreferrer">http://www.foxrun.com.au/Products/Cylinders_with_Gadgets.aspx</a></p>
  <p><em><strong>sorry if i am not clear as you can see i am new to programming, the question i am asking is how can i get the images i get from the aspx page when i am clicking the next page button in the browser cause if i can only scrape one page cause the url is not changing unless i send a http post somehow to tell the page to show the next page with new pictures, because the url stays the same i hope i am clear</strong></em></p>