Take the 2-minute tour ×

I'm scraping a site, searching for JPGs to download.
Scraping the site's HTML pages works fine.
But when I try getting the JPGs with CURL, copy(), fopen(), etc., I get a 403 forbiden status.

I know that's because the site owners don't want their images scraped, so I understand a good answer would be just don't do it, because they don't want you to.

Ok, but let's say it's ok and I try to work around this, how could this be achieved?

If I get the same URL with a browser, I can open the image perfectly, it's not that my IP is banned or anything, and I'm testing the scraper one file at a time, so it's not blocking me because I make too many requests too often.

From my understanding, it could be that either the site is checking for some cookies that confirm that I'm using a browser and browsing their site before I download a JPG.
Or that maybe PHP is using some user agent for the requests that the server can detect and filter out.

Anyway, have any idea?

share|improve this question
Sorry, but the first answer you provided is the correct one: don't do it. –  Juhana Mar 29 '12 at 6:55
The question isn't "should I do it or not", it's "how can I do it". I really hate it when people sidestep questions because they want to feel ethically righteous. @Petruza, for a start, check out HTTP Referrer spoofing. You probably want to fully mimic browser headers in the long term though. –  Leigh Mar 29 '12 at 7:15
@Juhana: When did scraping a site become breaking the law? –  Petruza Mar 29 '12 at 7:59
Wikinews and most of the uk.gov website have a public policy saying the information can be re-used but privately make this almost imposibble to implement as an automated process. –  Skizz Mar 29 '12 at 17:27
@Leigh, you were right, the referer did it. Thanks! –  Petruza Mar 30 '12 at 16:41
add comment

3 Answers

Are you able to view the page through a browser? Wouldn't a simple search of the page source find all images?

`    $findme   = '.jpg';
$pos = strpos($html, $findme);

if ($pos === false) {
    echo "The string '$findme' was not found in the string '$html'";
} else {
    echo "Images found..

   ///grab image location code

} `
share|improve this answer
It has thousands of pages with thousands of images. –  Petruza Mar 30 '12 at 1:45
add comment

Basic image retrieval:

Using the GD Library plugin commonly installed by default with many web hosts. This is something of an ugly hack but some may find the fact it can be done this way useful.

$remote_img = 'http://www.somwhere.com/images/image.jpg';
$img = imagecreatefromjpeg($remote_img);
$path = 'images/';
imagejpeg($img, $path);

Classic cURL image grabbing function for when you have extracted the location of the image from the donor pages HTML.

function save_image($img,$fullpath){
$ch = curl_init ($img);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,1);
curl_close ($ch);
$fp = fopen($fullpath,'x');
fwrite($fp, $rawdata);

If the basic cURL image grabbing function fails then the donor site probably has some form of server side defences in place to prevent retrieval and so you are probably breaching the terms of service by proceeding further. Though rare some sites do create images 'on the fly' using the GD library module, so what may look like a link to an image is actually a PHP script and that could be checking for things like a cookie, referer or session value being passed to it before the image is created and outputted.

share|improve this answer
In my question I wrote that CURL is getting a 403 error, as well as fopen and copy. I'm not really sure, but I guess imagecreatefromjpeg() won't be much different. –  Petruza Mar 30 '12 at 1:47
Look at what the whole webpage is actually doing using Firebug & a good cookie manager. Curl can mimic any useragent (pretend to be a web browser) and has the ability to send & recieve cookies. –  Skizz Mar 30 '12 at 9:46
thanks, I just did it with Chrome, which is what I use, but has a built-in tool just like firebug. –  Petruza Mar 30 '12 at 16:44
add comment
up vote 1 down vote accepted

Actually it was quite simple.
As @Leigh suggested, it only took spoofing an http referrer with the option CURLOPT_REFERER.
In fact for every request, I just provided the domain name as the referrer and it worked.

share|improve this answer
add comment

Your Answer


