title: How to use SOCKS proxies to make requests with aiohttp?
body: <p>I am trying to use <code>aiohttp</code> to make asynchronous HTTP requests over multiple SOCKS proxies. Basically, I am creating a pool of Tor clients with different IP addresses, and want to be able to route HTTP requests through them using <code>aiohttp</code>.</p>
  
  <p>Based on the suggestions <a href="https://stackoverflow.com/questions/36960800/is-there-any-way-to-use-aiohttp-client-with-socks-proxy/39593429#comment87410374_39593429">here</a> and <a href="https://stackoverflow.com/questions/45511728/how-can-i-use-socks4-and-socks5-proxies-with-aiohttp-clientsession">here</a>, I have been trying to use <a href="https://github.com/nibrag/aiosocks" rel="nofollow noreferrer">aiosocks</a>, but the examples in those threads do not work (if they ever did) because they are based on an old version of <code>aiosocks</code> with a different API. Documentation and examples of using <code>aiosocks</code> online are very sparse (it doesn't seem widely used). But I haven't been able to find any other solutions for using <code>aiohttp</code> with SOCKS proxies.</p>
  
  <p>Below is the code I have so far (sorry for the large amount of code - I tried to slim down the example as much as I could!). First I initialize the Tor clients with <code>stem</code>:</p>
  
  <pre><code>from datetime import datetime
  import stem.process
  
  from TorUtils import printCircuits, cleanShutdown
  
  NUM_TOR_CLIENTS = 3
  
  # create list of (source_port, control_port) tuples
  tor_ports = [(str(9050 + i), str(9050 + NUM_TOR_CLIENTS + i)) for i in range(NUM_TOR_CLIENTS)]
  
  # Every ISO 3166 country code except for {US} and {CA}
  country_codes = '{AF}, {AX}, {AL}, {DZ}, {AS}, {AD}, {AO}, {AI}, {AQ}, {AG}, {AR}, {AM}, {AW}, {AU}, {AT}, {AZ}, {BS}, {BH}, {BD}, {BB}, {BY}, {BE}, {BZ}, {BJ}, {BM}, {BT}, {BO}, {BQ}, {BA}, {BW}, {BV}, {BR}, {IO}, {BN}, {BG}, {BF}, {BI}, {KH}, {CM}, {CV}, {KY}, {CF}, {TD}, {CL}, {CN}, {CX}, {CC}, {CO}, {KM}, {CG}, {CD}, {CK}, {CR}, {CI}, {HR}, {CU}, {CW}, {CY}, {CZ}, {DK}, {DJ}, {DM}, {DO}, {EC}, {EG}, {SV}, {GQ}, {ER}, {EE}, {ET}, {FK}, {FO}, {FJ}, {FI}, {FR}, {GF}, {PF}, {TF}, {GA}, {GM}, {GE}, {DE}, {GH}, {GI}, {GR}, {GL}, {GD}, {GP}, {GU}, {GT}, {GG}, {GN}, {GW}, {GY}, {HT}, {HM}, {VA}, {HN}, {HK}, {HU}, {IS}, {IN}, {ID}, {IR}, {IQ}, {IE}, {IM}, {IL}, {IT}, {JM}, {JP}, {JE}, {JO}, {KZ}, {KE}, {KI}, {KP}, {KR}, {KW}, {KG}, {LA}, {LV}, {LB}, {LS}, {LR}, {LY}, {LI}, {LT}, {LU}, {MO}, {MK}, {MG}, {MW}, {MY}, {MV}, {ML}, {MT}, {MH}, {MQ}, {MR}, {MU}, {YT}, {MX}, {FM}, {MD}, {MC}, {MN}, {ME}, {MS}, {MA}, {MZ}, {MM}, {NA}, {NR}, {NP}, {NL}, {NC}, {NZ}, {NI}, {NE}, {NG}, {NU}, {NF}, {MP}, {NO}, {OM}, {PK}, {PW}, {PS}, {PA}, {PG}, {PY}, {PE}, {PH}, {PN}, {PL}, {PT}, {PR}, {QA}, {RE}, {RO}, {RU}, {RW}, {BL}, {SH}, {KN}, {LC}, {MF}, {PM}, {VC}, {WS}, {SM}, {ST}, {SA}, {SN}, {RS}, {SC}, {SL}, {SG}, {SX}, {SK}, {SI}, {SB}, {SO}, {ZA}, {GS}, {SS}, {ES}, {LK}, {SD}, {SR}, {SJ}, {SZ}, {SE}, {CH}, {SY}, {TW}, {TJ}, {TZ}, {TH}, {TL}, {TG}, {TK}, {TO}, {TT}, {TN}, {TR}, {TM}, {TC}, {TV}, {UG}, {UA}, {AE}, {GB}, {UM}, {UY}, {UZ}, {VU}, {VE}, {VN}, {VG}, {VI}, {WF}, {EH}, {YE}, {ZM}, {ZW}'
  
  tor_configs = [{'SOCKSPort': p[0], 'ControlPort': p[1], 'DataDirectory': './.tordata' + p[0], 
      'CookieAuthentication' : '1',  'MaxCircuitDirtiness': '3600', 'ExcludeNodes': country_codes,
      'EntryNodes': '{us}, {ca}', 'ExitNodes': '{us}, {ca}', 'StrictNodes': '1',
      'GeoIPExcludeUnknown': '1', 'EnforceDistinctSubnets': '0'
      } for p in tor_ports]
  
  print(f"Spawning {NUM_TOR_CLIENTS} tor clients ...")
  start_time = datetime.now()
  
  tor_clients = []
  for cfg in tor_configs:
      tor_clients.append({'config': cfg, 'process': stem.process.launch_tor_with_config(config = cfg)})
  </code></pre>
  
  <p>... and then I am trying to use the following code to make the HTTP requests with <code>aiohttp</code>:</p>
  
  <pre><code>from collections import defaultdict, deque
  from datetime import datetime, timedelta
  import asyncio
  import aiohttp
  import aiosocks
  from aiosocks.connector import ProxyConnector, ProxyClientRequest
  import async_timeout
  
  TIMEOUT = 10
  
  async def _get(url, session, proxy, request_limiter):
      try:
          async with request_limiter: # semaphore to limit number of concurrent requests
              async with async_timeout.timeout(TIMEOUT):
                  async with session.get(url, proxy=proxy, proxy_auth=None) as resp:
  
                      status = int(resp.status)
                      headers = dict(resp.headers)
                      content_type = str(resp.content_type)
                      text = await resp.text()
  
                      return {'url': url, 'status': status, 'headers': headers, 'text': str(text), 'errors': None}
  
      except asyncio.TimeoutError as e:
          queue.visited_urls[url] = datetime.now()
          return {'url': url, 'status': None, 'headers': None, 'text': None, 'errors': str(e)}
  
  
  async def _getPagesTasks(url_list, tor_clients, request_limiter, loop):
      """Launch requests for all web pages."""
  
      #deque rotates continuously through SOCKS sessions for each tor client ...
      sessions = deque()
      for tor_client in tor_clients:
          conn = ProxyConnector()
          session = aiohttp.ClientSession(connector=conn, request_class=ProxyClientRequest)
          sessions.append({'proxy': 'http://127.0.0.1:' + tor_client['config']['SOCKSPort'], 'session': session})
  
      tasks = []
      task_count = 0
      for url in url_list:
          s = sessions.popleft();
          session = s['session']
          proxy = s['proxy']
          task = loop.create_task(_get(url, session, proxy, request_limiter))
          tasks.append(task)
          task_count += 1
          session.append(s)
  
      results = await asyncio.gather(*tasks)
  
      for s in sessions:
          s.close()
  
      return results
  
  def getPages(url_list, tor_clients):
      """Given a URL list, dispatch pool of tor clients to concurrently fetch URLs"""
  
      request_limiter = asyncio.Semaphore(len(tor_clients)) # limit to one request per client at a time
  
      loop = asyncio.new_event_loop()
      asyncio.set_event_loop(loop)
      responses = loop.run_until_complete(_getPagesTasks(url_list, tor_clients, request_limiter, loop))
  
      loop.close()
      return responses
  </code></pre>
  
  <p>This code is not running, however. When I try to run it, I get the error below. I'm wondering if I'm doing something wrong, or if this is some problem with <code>aiosocks</code> (which seems like it's been unmaintained for a while, and might be targetting an older version of <code>aiohttp</code> or something ...):</p>
  
  <pre><code>~/Code/gis project/code/TorGetQueue.py in _getPagesTasks(url_list, tor_clients, request_limiter, loop)
       50     sessions = deque()
       51     for client in tor_clients:
  ---&gt; 52         conn = ProxyConnector()
       53         session = aiohttp.ClientSession(connector=conn, request_class=ProxyClientRequest)
       54         sessions.append({'proxy': 'http://127.0.0.1:' + client['config']['SOCKSPort'], 'session': session})
  
  ~/.local/share/virtualenvs/code-pIyQci_2/lib/python3.6/site-packages/aiosocks/connector.py in __init__(self, verify_ssl, fingerprint, resolve, use_dns_cache, family, ssl_context, local_addr, resolver, keepalive_timeout, force_close, limit, limit_per_host, enable_cleanup_closed, loop, remote_resolve)
       54             force_close=force_close, limit=limit,  loop=loop,
       55             limit_per_host=limit_per_host, use_dns_cache=use_dns_cache,
  ---&gt; 56             enable_cleanup_closed=enable_cleanup_closed)
       57 
       58         self._remote_resolve = remote_resolve
  
  TypeError: __init__() got an unexpected keyword argument 'resolve'
  </code></pre>
  
  <p>What am I doing wrong here? Is there an easier way to use SOCKS proxies with <code>aiohttp</code>? What do I need to change to make this code work with <code>aiosocks</code>?</p>
  
  <p>Thanks!</p>