the present invention in one embodiment includes a computer implemented method for performing a crawl of a web - site , that contains linked web pages . the invention includes retrieving a cookie corresponding to the root web page and retrieving a web page that is linked to said root web page by utilizing said cookie corresponding to said root web page to gain access to said hyperlinked web page .