public boolean isParseRobotsTxt(){
  return parseRobotsTxt;
}

public boolean isParseRobotsTxt ( ) { return parseRobotsTxt ; }
checks if the spider should parse the robots . txt for uris ( not related to following the directions ) .
