PHP cURL sitemap generator <p>Hi All I am having a problem with my sitemap generator it works great on a few thousand URLS but when it gets to 20000 URL it starts to go wrong is there anything else i can be doing to help prevent this </p> <pre><code>$request_url ="http://www.example.com";//put your url here $url=str_replace("http://", "", $request_url); // i added this because to check if wwwdomain.com is in the url $alllinks=array();// create an array for all the links $alllinks2=array();// create an array for all the links $alllinks3=array();// create an array for all the links $newUrl=array();// create an array for all the links $badUrls=array("basket", "#" ,"mailto", "javascript:document", "reviews.php", "review.php","tab=",".JPG",".jpg","png","PNG","gif","GIF","item_id=","../", "pn_pr");// array of urls we dont want to include $ch = curl_init(); //search for links curl_setopt($ch, CURLOPT_USERAGENT, $userAgent); curl_setopt($ch, CURLOPT_URL, $request_url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt ($ch,CURLOPT_VERBOSE,false); curl_setopt($ch, CURLOPT_TIMEOUT, 900); $result = curl_exec($ch); </code></pre> <p>Thank you</p>