title: cURL Scrape then Parse/Find Specific Content
body: <p>I'm using php and cURL to scrape a web page, but this web page is poorly designed (as in no classes or ids on tags), so I need to search for specific text, then go to the tag holding it (ie <code>&lt;p&gt;</code>) then move to the next child (or next <code>&lt;p&gt;</code>) and get the text. </p>
  
  <p>There are various things I need to get from the page, some also being the text within an <code>&lt;a onclick="get this stuff here"&gt;</code>. So basically I feel that I need to use cURL to scrape the source code to a php variable, then I can use php to kind of parse through and find the stuff I need. </p>
  
  <p>Does this sound like the best method to do this? Does anyone have any pointers or can demonstrate how I can put source code from cURL into a variable? </p>
  
  <p>Thanks!</p>
  
  <p>EDIT (Working/Current Code) -----------</p>
  
  <pre><code>&lt;?php
  
  class Scrape
  {
  public $cookies = 'cookies.txt';
  private $user = null;
  private $pass = null;
  
  /*Data generated from cURL*/
  public $content = null;
  public $response = null;
  
  /* Links */
  private $url = array(
                      'login'      =&gt; 'https://website.com/login.jsp',
                      'submit'     =&gt; 'https://website.com/LoginServlet',
                      'page1'      =&gt; 'https://website.com/page1',
                      'page2'      =&gt; 'https://website.com/page2', 
                      'page3'      =&gt; 'https://website.com/page3'
                      );
  
  /* Fields */
  public $data = array();
  
  public function __construct ($user, $pass)
  {
  
      $this-&gt;user = $user;
      $this-&gt;pass = $pass;
  
  }
  
  public function login()
  {
  
              $this-&gt;cURL($this-&gt;url['login']);
  
              if($form = $this-&gt;getFormFields($this-&gt;content, 'login'))
              {
                  $form['login'] = $this-&gt;user;
                  $form['password'] =$this-&gt;pass;
                  // echo "&lt;pre&gt;".print_r($form,true);exit;
                  $this-&gt;cURL($this-&gt;url['submit'], $form);
                  //echo $this-&gt;content;//exit;
              }
             //echo $this-&gt;content;//exit;
  }
  
  // NEW TESTING
  public function loadPage($page)
  {
              $this-&gt;cURL($this-&gt;url[$page]);
              echo $this-&gt;content;//exit;
  }
  
  /* Scan for form */
  private function getFormFields($data, $id)
  {
          if (preg_match('/(&lt;form.*?name=.?'.$id.'.*?&lt;\/form&gt;)/is', $data, $matches)) {
              $inputs = $this-&gt;getInputs($matches[1]);
  
              return $inputs;
          } else {
              return false;
          }
  
  }
  
  /* Get Inputs in form */
  private function getInputs($form)
  {
      $inputs = array();
  
      $elements = preg_match_all('/(&lt;input[^&gt;]+&gt;)/is', $form, $matches);
  
      if ($elements &gt; 0) {
          for($i = 0; $i &lt; $elements; $i++) {
              $el = preg_replace('/\s{2,}/', ' ', $matches[1][$i]);
  
              if (preg_match('/name=(?:["\'])?([^"\'\s]*)/i', $el, $name)) {
                  $name  = $name[1];
                  $value = '';
  
                  if (preg_match('/value=(?:["\'])?([^"\']*)/i', $el, $value)) {
                      $value = $value[1];
                  }
  
                  $inputs[$name] = $value;
              }
          }
      }
  
      return $inputs;
  }
  
  /* Perform curl function to specific URL provided */
  public function cURL($url, $post = false)
  {
      $ch = curl_init();
      curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
      curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13");
          // "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36");
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
      curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
      curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
      curl_setopt($ch, CURLOPT_VERBOSE, 1);
      curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
      curl_setopt($ch, CURLOPT_COOKIEJAR, $this-&gt;cookies);
      curl_setopt($ch, CURLOPT_COOKIEFILE, $this-&gt;cookies);
      curl_setopt($ch, CURLOPT_HEADER, 0);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
      curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 120);
      curl_setopt($ch, CURLOPT_TIMEOUT, 120);
  
      if($post)   //if post is needed
      {
          curl_setopt($ch, CURLOPT_POST, 1);
          curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post));
      }
  
      curl_setopt($ch, CURLOPT_URL, $url);
      $this-&gt;content = curl_exec($ch);
      $this-&gt;response = curl_getinfo( $ch );
      $this-&gt;url['last_url'] = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
      curl_close($ch);
  }
  }
  
  
  $sc = new Scrape('user','pass');
  $sc-&gt;login();
  
  $sc-&gt;loadPage('page1');
  echo "&lt;h1&gt;TESTTESTEST&lt;/h1&gt;";
  
  $sc-&gt;loadPage('page2');
  echo "&lt;h1&gt;TESTTESTEST&lt;/h1&gt;";
  
  $sc-&gt;loadPage('page3');
  echo "&lt;h1&gt;TESTTESTEST&lt;/h1&gt;";
  </code></pre>
  
  <p>(note: credit to @Ramz <a href="https://stackoverflow.com/questions/20255725/scrape-a-website-with-secured-login">scrape a website with secured login</a>)</p>