title: Python + Selenium firefox webdriver - pulling out images out of a website
body: <p>I am trying to pull out images from a webpage using:
  Python 2.7 + Selenium (using FireFox) + Beautiful Soup.</p>
  
  <p>The page loads dynamically, hence, I'm using Selenium for screen scraping.  Everything looks great on the front end, however, when all the images I loaded, and I look at the HTML, I can't see the links to the images.  Any ideas what could be going on here?</p>
  
  <p>Site is <a href="https://flipp.com/flyers?postal_code=97035" rel="nofollow noreferrer">https://flipp.com/flyers?postal_code=97035</a> ,
  and from there, navigate to <a href="https://flipp.com/weekly_ad/1550082-big-5-sporting-goods-weekly-ad" rel="nofollow noreferrer">https://flipp.com/weekly_ad/1550082-big-5-sporting-goods-weekly-ad</a> in order to see the first weekly ad (My working Code is below).</p>
  
  <p>To make things even more weird, I'm able to see that the images ARE loading in the inspector window... But I still can't see them in the HTML.  Any idea on whats going on here, and how to grab the updated HTML (after images load?)</p>
  
  <p>Here is the set of images i am able to pull from HTML (by appending jpg).  These are just for popup windows for when you hover over the canvas.  </p>
  
  <p><a href="https://i.stack.imgur.com/dpzlU.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/dpzlU.jpg" alt="enter image description here"></a></p>
  
  <p>What I am trying to get to are actually the images that make up the actual pages/canvas.  I can see them come through (using traffic option in firefox), but they are not appearing in HTML for some reason.  Any idea whats going on here?</p>
  
  <p><a href="https://i.stack.imgur.com/TNMxR.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/TNMxR.jpg" alt="enter image description here"></a></p>
  
  <p>Working code:</p>
  
  <pre><code>#import packages
  from time import gmtime, strftime,sleep, time
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium import webdriver
  from selenium.webdriver.common.proxy import Proxy, ProxyType
  from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
  #scraping packages
  from bs4 import BeautifulSoup
  
  
  USAPROXY = "177.84.23.122:3128"
  def launch_webdriver(PROXY):
      PROXY = PROXY
      PROXY_HOST = PROXY.rpartition(':')[0]
      PROXY_PORT = PROXY.rpartition(':')[2]
      fp = webdriver.FirefoxProfile()
      # Direct = 0, Manual = 1, PAC = 2, AUTODETECT = 4, SYSTEM = 5
      fp.set_preference("network.proxy.type", 1)
      fp.set_preference("network.proxy.http",PROXY_HOST)
      fp.set_preference("network.proxy.http_port",int(PROXY_PORT))
      fp.set_preference("network.proxy.ssl",PROXY_HOST)
      fp.set_preference("network.proxy.ssl_port",int(PROXY_PORT))
      fp.set_preference("general.useragent.override","whater_useragent")    
      fp.update_preferences()
      return webdriver.Firefox(firefox_profile=fp)
  
  
  
  
  def test():
      driver = launch_webdriver(USAPROXY)
      driver.set_page_load_timeout(11)
      driver.get("https://flipp.com/flyers?postal_code=97035")
      sleep(15)
      driver.get("https://flipp.com/weekly_ad/1550082-big-5-sporting-goods-weekly-ad")
      sleep(5)
      my_html = driver.page_source
      soup = BeautifulSoup(my_html,'lxml')
      tags=soup.findAll('img')  #prints only 3 imgs, there should be 100s
      for tag in tags:print tag
      print soup.prettify()
  #execute script
  test()
  </code></pre>