a method for incremental crawling of content stored on a plurality of content providers using aggregation is provided . the method comprises receiving a request to crawl content on one or more associated content providers ; retrieving one or more first references to content on a first content provider ; retrieving one or more second references to content on one or more second content providers during the same request ; aggregating the first and second references ; and returning the aggregated first and second references . this is done while taking into consideration opaque timestamp object which is managed in a distributed manner . the opaque timestamp is filled in by the content providers but stored in the crawler side between crawling sessions .