title: Get the pages source using php and python
body: <p>I'm running into an issue while web scraping. I'm using PHP for scraping, but there's this certain website where I have to use Python instead. My solution was to call a Python script from PHP to get the page source, as my main script must be written in PHP. </p>
  
  <p>The Python code below works great when I call it using <em>Python</em>:</p>
  
  <pre><code>from selenium import webdriver
  from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
  driver = webdriver.Remote("http://127.0.0.1:4444/wd/hub", DesiredCapabilities.CHROME)
  link = 'websitelink'
  driver.get(link)
  s = driver.page_source
  print((s.encode("utf-8")))
  driver.quit()
  </code></pre>
  
  <p>The Python code below is what I use within the PHP script, because I'm passing the website URL to it as a variable:</p>
  
  <pre><code>import sys
  link = sys.argv[1]
  from selenium import webdriver
  from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
  driver = webdriver.Remote("http://127.0.0.1:4444/wd/hub", DesiredCapabilities.CHROME)
  driver.get(link)
  s = driver.page_source
  print((s.encode("utf-8")))
  driver.quit()
  </code></pre>
  
  <p>This is the PHP code where i call the Python script:</p>
  
  <pre><code>$page_html =  shell_exec('python3.6 test.py '.$url);
  </code></pre>
  
  <p>So the intended result is that <code>page_html</code> will have the page content from the website.</p>
  
  <p>The problem is that my website URL has page 1, page2, page 3... and in PHP I only receive the first page, yet when I use just the Python script I can get the correct results. I lost 2 days of my life on this; does someone have an idea why I don't receive the entire page contents when I start the Python script from PHP?</p>