title: How to automate web scraping using php?
body: <p>Let me explain my situation.</p>
  
  <p>I have a list of 10 million page urls. I want those pages scraped and saved in the database as raw html.</p>
  
  <p>As of now I'm using CURL to scrap those pages. Every time I access <code>index.php</code>, it scraps one page url and save it in the database. </p>
  
  <p>Now I think its not possible to access that <code>index.php</code> 10 million times by myself using browser.  </p>
  
  <p>I could use <code>do while loop</code>. But I think its gonna take hell of a time to complete the task.  And memory issues too.</p>
  
  <p>So can someone point me in the right direction to make this task painless.</p>
  
  <p>I own a Linux VPS server with <code>1GB RAM</code> and WHM/cPanel.</p>
  
  <p>PS: I considered CRON. But i have to define the time in cron. Even if I run the script every 1 minute using cron, I can only complete <code>1440 urls in 24 hours</code>.  So can some one give me some idea to complete <code>atleast 100,000</code> urls in one day using cron?.  </p>