lib/wmap/url_crawler.rb
# Pre-crawl profiler, to be used for network profiling to maximum the crawler performance.
Pre-crawl profiler, to be used for network profiling to maximum the crawler performance.
Pre - crawl profiler to be used for network profiling to maximum the crawler performance .
def pre_crawl(url)
		begin
			puts "Perform network profiling works on the web server before the web crawling: #{url}" if @verbose
			host=url_2_host(url)
			nwk_to=Wmap::NetworkProfiler.new.profile(host).to_i
			if (1500 + Wmap::NetworkProfiler.new.profile(host)*2).to_i > Max_http_timeout
				@http_timeout = Max_http_timeout
			else
				@http_timeout = 1500 + nwk_to*2
			end
			puts "Done with the pre-scan works: reset @http_timeout to: #{@http_timeout} ms" if @verbose
		rescue Exception => ee
			puts "Exception on method #{__method__} for #{host}: #{ee}" if @verbose
			@http_timeout = Max_http_timeout
		end
	end
def pre_crawl ( url ) begin puts "Perform network profiling works on the web server before the web crawling: #{url}" if @verbose host = url_2_host ( url ) nwk_to = Wmap :: NetworkProfiler . new . profile ( host ) . to_i if ( 1500 + Wmap :: NetworkProfiler . new . profile ( host ) * 2 ) . to_i > Max_http_timeout @http_timeout = Max_http_timeout else @http_timeout = 1500 + nwk_to * 2 end puts "Done with the pre-scan works: reset @http_timeout to: #{@http_timeout} ms" if @verbose rescue Exception => ee puts "Exception on method #{__method__} for #{host}: #{ee}" if @verbose @http_timeout = Max_http_timeout end end
Pre-crawl profiler, to be used for network profiling to maximum the crawler performance.
Pre - crawl profiler to be used for network profiling to maximum the crawler performance .
# Use the following formula to 'guess' the right http time-out threshold for the scanner
url
<not_specific>