title: Set the Crawler rules dynamically in python
body: <p>I am setting up a crawler using Scrapy and want to be able to setup the rule dynamically based on a variable in the spider. The code I have is something like this:</p>
  
  <pre><code>class MainSpider(CrawlSpider):
    def __init__(self, configuration):
      super(MainSpider, self).__init__(configuration.name)
      dispatcher.connect(self.spider_closed, signals.spider_closed)
      self.product_link_id = configuration.product_link_id
  
    rules = (
      # call parse_link on all links from starting url
      Rule(LinkExtractor(allow=product_link_id), callback='parse_link', follow=False),)
  </code></pre>
  
  <p>As you can see, I'm trying to use the instance variable there, which doesn't work. I am fairly certain this is because of a fundamental Python issue with my code that has nothing to do with Scrapy but not sure what it is as I'm new to Python.</p>