lib/spiderman/crawler.rb
# Adds a url to the hash of queued urls
# Does not add the url if
#   * crawl_depth has exeeded max crawl depth + 1 (allowed to scrape and store links for next level)
#   * url is already in queue
#   * url is already crawled
#   * url has been tried and failed
#
#   Returns truthy if added to queue
Adds a url to the hash of queued urls
Does not add the url if
crawl_depth has exeeded max crawl depth + 1 (allowed to scrape and store links for next level)
url is already in queue
url is already crawled
url has been tried and failed

Returns truthy if added to queue
Adds a url to the hash of queued urls Does not add the url if crawl_depth has exeeded max crawl depth + 1 ( allowed to scrape and store links for next level ) url is already in queue url is already crawled url has been tried and failed Returns truthy if added to queue
def queue_url(url_id, url_string, crawl_depth)
      return if (crawl_depth > @opts[:max_crawl_depth]+1 ||
                 @n_discovered_urls >= @opts[:max_urls] ||
                 url_discovered?(url_id) )
      @n_discovered_urls += 1
      @queued_urls[url_id] = {url_string: url_string, crawl_depth: crawl_depth}
      puts "#{url_id}, #{crawl_depth}, #{@n_discovered_urls}, #{@queued_urls.size}, #{@crawled_urls.size}, #{@failed_urls.size}, #{url_string}"
    end
def queue_url ( url_id , url_string , crawl_depth ) return if ( crawl_depth > @opts [ :max_crawl_depth ] + 1 || @n_discovered_urls >= @opts [ :max_urls ] || url_discovered? ( url_id ) ) @n_discovered_urls += 1 @queued_urls [ url_id ] = { url_string : url_string , crawl_depth : crawl_depth } puts "#{url_id}, #{crawl_depth}, #{@n_discovered_urls}, #{@queued_urls.size}, #{@crawled_urls.size}, #{@failed_urls.size}, #{url_string}" end
Adds a url to the hash of queued urls
Does not add the url if
 crawl_depth has exeeded max crawl depth + 1 (allowed to scrape and store links for next level)
 url is already in queue
 url is already crawled
 url has been tried and failed
Adds a url to the hash of queued urls Does not add the url if crawl_depth has exeeded max crawl depth + 1 ( allowed to scrape and store links for next level ) url is already in queue url is already crawled url has been tried and failed
url_id
<not_specific>
url_string
<not_specific>
crawl_depth
<not_specific>