title: callback to parse function in scrapy class
body: <p>I've been trying to make a Discord 'bot' of sorts with an integrated Scrapy spider that scrapes data from a website (that has no APIs) and outputs the parsed data depending on the command sent on Discord to the bot.</p>
  <p>I've managed to nail down the scraping part as I can get the data I need from a list array and output it in a filesystem file with Scrapy's commands:</p>
  <pre class="lang-py prettyprint-override"><code>import scrapy
  
  class locg(scrapy.Spider):
  
      name = 'spiderbot'
      start_urls = ['https://leagueofcomicgeeks.com/comics/new-comics']
  
      def start_requests(self):
          headers= {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'}
          for url in self.start_urls:
                  yield scrapy.Request(url, headers=headers)
  
      def parse(self, response):
          for items in response.xpath('.//ul[@class=&quot;comic-list-thumbs item-list-thumbs item-list&quot;]'):
  
              publishers = items.xpath('.//div[@class=&quot;publisher color-offset&quot;]/text()').extract()
              issues = items.xpath(&quot;.//div[@class='title color-primary']//a/text()&quot;).extract()
  
              i = 0
              for publisher in publishers:
                  if &quot;Marvel Comics&quot; in publisher:
                      print(issues[i])
                  i = i+1
  </code></pre>
  <p>With this code I can match in the list every issue name that is with publisher Marvel Comics.</p>
  <p>I changed the code to integrate it with the Discord python API, the problem is I can't call the <code>parse</code> function from the Scrapy class outside of it so that I can make sure it runs whenever a function inside the Discord class gets called by the discord command.</p>
  <p>So this is what I currently have:</p>
  <pre class="lang-py prettyprint-override"><code>import discord
  import os
  import scrapy
  
  marvel = []
  dc = []
  
  class locg(scrapy.Spider):
  
      name = 'spiderbot'
      start_urls = ['https://leagueofcomicgeeks.com/comics/new-comics']
  
      def start_requests(self):
          headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'}
          for url in self.start_urls:
                  yield scrapy.Request(url, headers=headers)
  
      def parse(self, response):
          for items in response.xpath('.//ul[@class=&quot;comic-list-thumbs item-list-thumbs item-list&quot;]'): 
  
              publishers = items.xpath('.//div[@class=&quot;publisher color-offset&quot;]/text()').extract()
              issues = items.xpath(&quot;.//div[@class='title color-primary']//a/text()&quot;).extract()
  
              i = 0
              for publisher in publishers:
                  if &quot;Marvel Comics&quot; in publisher:
                      marvel.append(issues[i])
                  if &quot;DC Comics&quot; in publisher:
                      dc.append(issues[i])
                  i = i+1
  
  
  client = discord.Client()
  
  @client.event
  async def on_ready():
      print('We have logged in as {0.user}'.format(client))
  
  @client.event
  async def on_message(message):
      if message.author == client.user:
          return
  
      test = locg()
      test.parse()
  
      if message.content.startswith('-marvel'):
          await message.channel.send(marvel)
  
      if message.content.startswith('-dc'):
          await message.channel.send(dc)
  
  client.run('token')
  </code></pre>
  <p>When I run this and I call the command on my Discord server the bots just gives me <code>[]</code> so the empty list from the global variable I defined at the start of the code. Which makes me think the Scrapy class isn't running when I call the command and so the variable is empty.</p>
  <p>I tried to instanciate the class with</p>
  <pre class="lang-py prettyprint-override"><code>    test = locg()
      test.parse()
  </code></pre>
  <p>But I get <code>No value for argument 'response' in method call</code> and I'm not really sure how to define the response value in here.</p>
  <p>Does anyone have any pointers into this?
  I realize that maybe this isn't really what Scrapy was designed to do so please do tell me if so, I'll look for other ways to do this.</p>
  <p>I appreciate any and all help, please let me know if I should give more info or if I should've done anything different with my first question. This is my first day on the website.</p>
  <p>Thank you.</p>