title: Running all instances of inherited classes with one method
body: <p>I'm trying to build an asyncronous parser. I'm a novice in both OOP and asyncronous programming.</p>
  <p>I have 3 main resources from which I'm parsing data. Within each resource I have several web pages to parse from, ranging from 3 to 30 pages. I would like to keep a persistent ClientSession() for each of the 3 main resources due to authorization procedures.</p>
  <p>I'm creating a main Parser class, three subclasses, and more 'sub-subclasses' which inherit from the three subclasses. The idea is that I run a single Parser.run() method, and all three resources get parsed automatically, returning one json object.</p>
  <p>I need it in a class, because I'm planning to scale it so that I run many instances of the parent Parser class, each will have it's own authorization data and I'd like them all to run asynchronously.
  For now I have three main questions:</p>
  <ol>
  <li><p>How do I initialize all my child class instances, each with different url to parse, so that I can later run them all with one method?</p>
  </li>
  <li><p>I need to keep a persistent ClientSession() for each resource, am I doing it correctly?</p>
  </li>
  <li><p>How do I then return a json object (a dict) from each resource and access them all from the Parser() instance?</p>
  </li>
  </ol>
  <p>I would be grateful for any directions or links.</p>
  <pre><code>import asyncio
  from asyncio import Semaphore
  from datetime import datetime
  import random
  
  from aiohttp import ClientSession
  
  
  class Parser:
      # several user agents to randomly sample from
      user_agents = [
          'Mozilla/5.0 (Windows NT 6.3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 YaBrowser/17.6.1.749 Yowser/2.5 Safari/537.36',
          'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36',
          'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 YaBrowser/18.3.1.1232 Yowser/2.5 Safari/537.36',
          'Mozilla/5.0 (iPhone; CPU iPhone OS 10_0 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 YaBrowser/17.4.3.195.10 Mobile/14A346 Safari/E7FBAF',
          'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36']
  
      def __init__(self):
          self.url = None
          self.headers = {}
          self.user_agent = random.choice(Parser.user_agents)
          self.parsers = [] # keep three SupParsers (Api1Parser, Api2Parser, Api3Parser)
          self.response = None # keep response for logging purposes
  
      async def parse(self, session: ClientSession): # empty method for inheritance
          pass
  
      def log(self):
          print(datetime.now(), flush=True)
          print(self.url, flush=True)
          print(self.response.status_code, flush=True)
  
      def add_parser(self, parser: 'Parser'):
          self.parsers.append(parser)
  
      def run(self):
          loop = asyncio.get_event_loop()
          futures = []
          for parser in self.parsers:
              futures.append(asyncio.create_task(parser.gather_tasks()))
          all_futures = asyncio.gather(*futures)
          loop.run_until_complete(all_futures)
  
  
  class Api1Parser(Parser):
  
      semaphore = Semaphore(value=100) # limit total connections across each ClientSession()
  
      def __init__(self):
          super().__init__()
          self.headers['User-Agent'] = self.user_agent
          self.jobs = []
          self.tasks = []
          self.json_ = {}
          super().add_parser(self)
  
      def add_job(self, job: 'Parser'):
          self.jobs.append(job)
  
      async def gather_tasks(self):
          async with ClientSession(headers=self.headers) as session:
              for job in self.jobs:
                  task = asyncio.create_task(job.parse(session))
                  self.tasks.append(task)
          await asyncio.gather(*self.tasks)
  
  
  class Page1Api1Parser(Api1Parser):
      def __init__(self):
          super().__init__()
          self.url = 'https://someApiService.domain'
          super().add_job(self)
  
      async def parse(self, session: ClientSession):
          async with ApiParser.semaphore:
              async with session.get(self.url) as resp:
                  self.response = resp
                  json_ = await resp.read()
                  super().json_['Page1'] = json_
  
  
  parser = Parser()
  parser.run()
  </code></pre>