title: Selenium: difference between chrome and PhantomJS?-python
body: <p>I want to do web scraping for Bing's search results. Basically, I am using selenium, the idea is to using selenium to click 'Next' automatedly and scrap the URLs of search results of each page. I made it run with chrome browser on my Ubuntu:</p>
  
  <pre><code>from selenium import web driver
  import os
  
  class bingURL(object):
  
      def __init__(self):
          self.driver=webdriver.Chrome(os.path.expanduser('./chromedriver'))
  
      def get_urls(self,url):
          driver=self.driver
          driver.get(url)
          elems = driver.find_elements_by_xpath("//a[@href]")
          href=[]
          for elem in elems:
              link=elem.get_attribute("href") 
              try:
                  if 'bing.com' not in link and 'http' in link and 'microsoft.com' not in link and 'smashboards.com' not in link:
                      href.append(link)
              except:
                  pass
  
          return list(set(href))
  
  
  
  
      def search_urls(self,keyword,pagenum):
          driver=self.driver
          searchurl=self.lookup(keyword) ### url of first page of google search
          driver.get(searchurl)
          results=self.get_urls(searchurl) 
  
          for i in range(pagenum):
  
              driver.find_elements_by_class_name("sb_pagN")[0].click() # click 'Next' of bing search result
              time.sleep(5) # wait to load page
              current_url=driver.current_url
              #print(current_url)
              #print(self.get_urls(current_url))
              results[0:0]=self.get_urls(current_url)
  
          driver.quit()
          return results
  
  
  
  
  
  
      def lookup(self,query):
  
          return "https://www.bing.com/search?q="+query
  
  
  
  if __name__ == "__main__":
      g=bingURL()
      result=g.search_urls('Stackoverflow is good',10)
  </code></pre>
  
  <p>it works perfectly, when I run the code, it launches a Chrome browser, and I can saw it go to the next page automatically, and get URLs for 10 pages of searching results.</p>
  
  <p>However, my goal is to run these codes on AWS successfully. The original codes failed with error 'Chrome failed to start'. After google, it seems I need to use a headless browser like PhantomJS on AWS. Thus I installed PhantomJS, and change the <code>def __init__(self):</code> to:</p>
  
  <pre><code> def __init__(self):
          self.driver=webdriver.PhantomJS() 
  </code></pre>
  
  <p>However, it cannot click 'next' anymore, and cannot scrap URLs using the old code. The error message is:</p>
  
  <pre><code> File ".../SEARCH_BING_MODULE.py", line 70, in search_urls
      driver.find_elements_by_class_name("sb_pagN")[0].click() 
  
  IndexError: list index out of range
  </code></pre>
  
  <p>It looks like change the browser completely change the rules. How should I modify the more original code to make it work again? or how to scrap Bing search results' URLs using selenium+PhantomJS?</p>
  
  <p>Thanks for your help!</p>