title: PHP Curl Page Scraper Pagination Issue
body: <p>I am trying to scrape a site for listings using Simple HTML Dom and Curl. I can connect to the site and find the listings I require using simple htmldom, however, I am getting in to trouble when the results are paginated. I have found the source for the "next" link and I am feeding in what I believe to be the correct address, however, the function I wrote to parse the results is producing a copy of the first page.
  So in short, I am echoing out the same page twice. I am studying the whole post/GET process in Fiddler and as far as I can see, the headers in the GET  call for the paginated page are correct.</p>
  
  <p>Can anyone spot where it is I am going wrong ?.  </p>
  
  <pre><code> &lt;!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"&gt;
      &lt;html xmlns="http://www.w3.org/1999/xhtml"&gt;
      &lt;head&gt;
      &lt;meta http-equiv="Content-Type" content="text/html; charset=utf-8" /&gt;
      &lt;title&gt;Chelmsford Planning Applications&lt;/title&gt;
      &lt;/head&gt;
  
      &lt;body&gt;
     &lt;?php
  require_once('simple_html_dom.php');
  ?&gt;
  
  &lt;?php
  $html = new simple_html_dom();
  error_reporting(E_ALL);
  $cookie_file = "cookies/cookiejar.txt";
  
  $page = "http://publicaccess.chelmsford.gov.uk/online-applications/advancedSearchResults.do?action=firstPage";
  
  // Form fields are the fields being posted by the search form.          
  $form_fields = "searchCriteria.reference=&amp;searchCriteria.planningPortalReference=&amp;searchCriteria.alternativeReference=&amp;searchCriteria.description=&amp;searchCriteria.applicantName=&amp;searchCriteria.caseType=&amp;searchCriteria.ward=&amp;searchCriteria.parish=&amp;searchCriteria.agent=&amp;searchCriteria.caseStatus=&amp;searchCriteria.caseDecision=&amp;searchCriteria.appealStatus=&amp;searchCriteria.appealDecision=&amp;searchCriteria.developmentType=&amp;caseAddressType=Application&amp;searchCriteria.address=&amp;date%28applicationReceivedStart%29=08%2F11%2F2013&amp;date%28applicationReceivedEnd%29=18%2F11%2F2013&amp;date%28applicationValidatedStart%29=&amp;date%28applicationValidatedEnd%29=&amp;date%28applicationCommitteeStart%29=&amp;date%28applicationCommitteeEnd%29=&amp;date%28applicationDecisionStart%29=&amp;date%28applicationDecisionEnd%29=&amp;date%28appealDecisionStart%29=&amp;date%28appealDecisionEnd%29=&amp;date%28applicationDeterminedStart%29=&amp;date%28applicationDeterminedEnd%29=&amp;searchType=Application";
  
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_COOKIESESSION, true);
  curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
  curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_HEADER, true);
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_ENCODING, "");
  curl_setopt($ch, CURLOPT_AUTOREFERER, true);
  curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)');
  curl_setopt($ch, CURLOPT_URL, $page);
  curl_setopt($ch, CURLOPT_POST, true);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $form_fields);
  curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
  echo 'Curl error: ' . curl_error($ch); //no errrors 
  $str = curl_exec($ch);
  curl_close($ch);
  
  //Simple HTMLDOM Here....
  $html = str_get_html($str);
  $ret  = $html-&gt;find('ul li.searchresult');
  foreach ($ret as $list) {
      echo $list;
  }
  //If we find a NEXT button
  if ($next = $html-&gt;find('a[class=next]', 0)) {
      //We apend the next HREF to the base referer  
      $base = "http://publicaccess.chelmsford.gov.uk";
      $url  = $base . $next-&gt;href;
  
      // Display the Url to see if we are going to the right address.
      echo $url;
  
      //If we find a next link, we call the get_data function to follow the link 
      //and search for the same type of data to display.
      get_data($url);
  }
  ?&gt;
  
   &lt;?php
  function get_data($url)
  {
      $new_page = $url;
  
      $ch = curl_init();
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
      curl_setopt($ch, CURLOPT_HEADER, true);
      curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // follow redirects
      curl_setopt($ch, CURLOPT_ENCODING, ""); // handle all encodings
      curl_setopt($ch, CURLOPT_AUTOREFERER, true); // set referer on redirect
      curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)');
      curl_setopt($ch, CURLOPT_URL, $new_page); // This is the address of the paginated page
      $str1 = curl_exec($ch);
      curl_close($ch);
  
      $html = str_get_html($str1);
      # get all elements with class="searchresult"                           
      $ret  = $html-&gt;find('ul li.searchresult');
      foreach ($ret as $list) {
          echo $list;
  
      }
      // Display the Url again just to make sure our function is being called.
      echo "the new page is " . $new_page;
  }
  ?&gt;
  
      &lt;/body&gt;
      &lt;/html&gt;
  </code></pre>