title: how to run a python web scraper in the background of php
body: <p>I have been trying to figure out how to run a python web scraper from a PHP interface. The web scraper puts data into a mysql database, and the PHP interface echos the data in a table. I have spent a lot of time researching this, however the data never shows up in the table, even when I keep the data in the database, and reload the interface, no data shows up. Also When i call the python script in the terminal and then open the interface, the data arrives. Is there any way people could recommend doing this?
  the first php script is as follows:</p>
  
  <pre><code>&lt;!DOCTYPE HTML&gt;
  &lt;html&gt;
  &lt;body&gt;
  &lt;?php
  echo "3h test";
  exec("python 3h.py");
  ?&gt;
  &lt;form&gt;
  &lt;input type="button" value="ViewCode" onclick="location="'python 3h.py'&gt;
  &lt;/form&gt; 
  &lt;a href="action.php"&gt;Run script.&lt;/a&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  
  <p>the python crawler is:</p>
  
  <pre><code>from urllib import urlopen
  from bs4 import BeautifulSoup
  import pymysql.cursors
  
  # Webpage connection
  page1 = urlopen("http://www.officialcharts.com/charts/singles-    chart/19800203/7501/")
  page2= urlopen("http://www.officialcharts.com/charts/")
  # Grab title-artist classes and store in recordList
  bsObj0 = BeautifulSoup(page1,"html.parser")
  bsObj1 = BeautifulSoup(page2,"html.parser")
  
  recordList0 = bsObj0.findAll("div", {"class" : "title-artist",})
  
  recordList = recordList0 + bsObj1.findAll("div", {"class" : "title-artist",})
  connection = pymysql.connect(host='localhost',
                           user='root',
                           password='******',
                           db='crawler database',
                           charset='utf8mb4',
                           cursorclass=pymysql.cursors.DictCursor)
  
  try:
      with connection.cursor() as cursor:
          for record in recordList:
              title = record.find("div", {"class": "title",}).get_text().strip()
              artist = record.find("div", {"class": "artist"}).get_text().strip()
              sql = "INSERT INTO `mytable` (`title`,`artist`) VALUES (%s,%s)"
              cursor.execute(sql,(artist,title))
      connection.commit()
  finally:
      connection.close()
  </code></pre>
  
  <p>the python script is named 3h.py</p>
  
  <p>the final php script is to long for me to bother typing out, this is the core:</p>
  
  <pre><code> &lt;?php
   $no     = 1;
   $total  = 0;
   while ($row = mysqli_fetch_array($query))
   {
   echo '&lt;tr&gt;
   &lt;td&gt;'.$no.'&lt;/td&gt;
   &lt;td&gt;'.$row['itemName'].'&lt;/td&gt;
   &lt;td&gt;'.$row['itemCost'].'&lt;/td&gt;
   &lt;/tr&gt;';
   $no++;
   }?&gt;
  </code></pre>