def set_url(self, url):
        self.url = url
        self.host, self.path = urlparse.urlparse(url)[1:3]
Sets the URL referring to a robots.txt file.
dataset/ETHPy150Open babble/babble/include/jython/Lib/robotparser.py/RobotFileParser.set_url