title: Bypassing Imperva bot detection with Scrapy. Any way possible?
body: <p>I'm trying to scrape several links containing information about events. I am rotating my paid proxies and user agents generated by UserAgent library. Imperva, which requires a US IP, is so sensitive that even it doesn't allow my browser event if I use a free US proxy!</p>
  <p>I asked this question in a scrapiping-releated Discord channel. Someone contacted me and said it is possible to bypass Imperva but he can't tell me how because he doesn't wan't me as a competitor in the ticket scraping market :(</p>
  <p>In addition to user agents and proxies, I tried to imitate the browser's succesful request headers but it didn't work. I just have 405s and 403s. I will try to scrape the event section but I couldn't even see a 200 response for any of the 27 links I have ( I added some below)</p>
  <p><strong>How do you think Imperva could be bypassed with Scrapy or Requests?</strong> It's also okay to recommend me an academic resource which I can study to develope my Scrapy skills.</p>
  <hr />
  <p><strong>Some of the links I'm trying to scrape</strong></p>
  <pre><code>https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=COL&amp;linkID=tktldr&amp;shopperContext=&amp;caller=appList&amp;appCode=
  https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=FOR&amp;linkID=tktldr&amp;shopperContext=&amp;caller=appList&amp;appCode=
  https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=AMP&amp;linkID=tktldr&amp;shopperContext=&amp;caller=&amp;appCode=
  https://budweisergardens.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONC&amp;linkID=global-labatt&amp;shopperContext=&amp;caller=&amp;appCode=
  https://pplcenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=global-allentown&amp;shopperContext=&amp;caller=&amp;appCode=
  https://ynottix.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EVENTS&amp;linkID=global-odu&amp;shopperContext=&amp;caller=&amp;appCode=
  </code></pre>
  <p><strong>My spider code</strong> which is comprised of a class to import my proxies from file and the spider code proper. I add my proxy as a meta value as said in <a href="https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware" rel="nofollow noreferrer">Scrapy documentation</a>. I use download delays:</p>
  <pre><code>import scrapy
  from scrapy import Request
  from random_user_agent.user_agent import UserAgent
  import random
  import pandas as pd
  
      class ProxyFunctions:
          (...)
      
      class AlexSpider(scrapy.Spider):
      
          name = 'alex'
          s = ProxyFunctions()
          s.prox_list_fixer()  #proxylerin bulunduğu txt'yi düzelip yeni bir txt oluşturdu.
          proxies = s.imp_proxies()
          def __init__(self):
              self.root = &quot;https://partnercarrier.com&quot;
              self.start_url = &quot;https://partnercarrier.com/PA/&quot;
              #self.initial_links = self.imp_links() dosyadan tüm linkler eklendiğinde kullanılacak
              user_agent_rotator = UserAgent(software_names=['chrome'], operating_systems=['windows', 'linux'])
              self.user_agents = user_agent_rotator.get_user_agents()
              #self.root_link = &quot;https://www.google.com&quot;
              self.UA_rand = random.choice(self.user_agents)['user_agent'] #User Agent set
              #self.UA_LIST = open(&quot;/home/draco/docs/scraping/scrapyyy/thomas/USER_AGENTS.txt&quot;,&quot;r&quot;) #manual UA importation from  text
          #dosyadaki proxy listesinden random proxy alır
          def imp_randp(self, path=&quot;/home/draco/docs/scraping/scrapyyy/thomas/proxies.txt&quot;):
              with open (path) as PROXIES:
                  lines = PROXIES.readlines()
                  return random.choice(lines).strip()
          #dosyadan linkleri alır
          
          def imp_links(self, path=&quot;/home/draco/docs/scraping/Selenium/inputs.csv&quot;):
                  x = pd.read_csv(path)
                  links = x['Url']
                  links = [i for i in links]
                  return links
          def start_requests(self):
              print(&quot;INITIAL REQUEST&quot;)
              links = self.imp_links()
      
              for link in links:
                  print(f&quot;---INFO: Requesting page=&gt; {link}&quot;)
                  proxy = random.choice(self.proxies)
                  #print(&quot;---INFO: Using proxy =&gt; &quot;, proxy)
                  h = {
                      'User-Agent': random.choice(self.user_agents)['user_agent'],
                      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 
                      'Accept-Encoding': 'gzip, deflate, br', 
                      'Accept-Language': 'tr-TR,tr;q=0.9,en-US;q=0.8,en;q=0.7',
                      'Cache-Control': 'max-age=0',
                      'Connection': 'keep-alive',
                      'Host' : link.split(&quot;/&quot;)[2],
                      'Sec-Fetch-Dest': 'document',
                      'Upgrade-Insecure-Requests': '1',
                      'Sec-Fetch-Mode': 'navigate',
                      'sec-ch-ua-platform': '&quot;Linux&quot;',
                      'sec-ch-ua' : '&quot; Not A;Brand&quot;;v=&quot;99&quot;, &quot;Chromium&quot;;v=&quot;99&quot;, &quot;Google Chrome&quot;;v=&quot;99&quot;',
                      
                      }
                  b = 'groupCode=CONC&amp;linkID=global-labatt&amp;shopperContext=&amp;caller=&amp;appCode='
                  yield Request(
                                      url = link ,
                                      callback = self.parse_gen,
                                      headers = {&quot;user-agent&quot;: random.choice(self.user_agents)['user_agent']},
                                      meta = {&quot;proxy&quot;: proxy},
                                      body = b,
                                      dont_filter= True  
                                      )
          def parse_gen(self, response):
              print(&quot;---INFO: General parser opened. PARSER1&quot;)
  </code></pre>
  <p><strong>My terminal Output:</strong></p>
  <pre><code>draco@draco:~/docs/scraping/scrapyyy/upwork$ scrapy crawl alex
  https://umasstix.evenue.net
  2022-03-20 20:23:01 [scrapy.utils.log] INFO: Scrapy 2.5.1 started (bot: upwork)
  2022-03-20 20:23:01 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.7.0, Python 3.8.10 (default, Nov 26 2021, 20:14:08) - [GCC 9.3.0], pyOpenSSL 22.0.0 (OpenSSL 1.1.1m  14 Dec 2021), cryptography 36.0.1, Platform Linux-5.13.0-35-generic-x86_64-with-glibc2.29
  2022-03-20 20:23:01 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
  2022-03-20 20:23:01 [scrapy.crawler] INFO: Overridden settings:
  {'AUTOTHROTTLE_ENABLED': True,
   'BOT_NAME': 'upwork',
   'CONCURRENT_REQUESTS_PER_DOMAIN': 14,
   'HTTPCACHE_ENABLED': True,
   'NEWSPIDER_MODULE': 'upwork.spiders',
   'SPIDER_MODULES': ['upwork.spiders']}
  2022-03-20 20:23:01 [scrapy.extensions.telnet] INFO: Telnet Password: 7f185fdb1347847f
  2022-03-20 20:23:01 [scrapy.middleware] INFO: Enabled extensions:
  ['scrapy.extensions.corestats.CoreStats',
   'scrapy.extensions.telnet.TelnetConsole',
   'scrapy.extensions.memusage.MemoryUsage',
   'scrapy.extensions.logstats.LogStats',
   'scrapy.extensions.throttle.AutoThrottle']
  2022-03-20 20:23:05 [scrapy.middleware] INFO: Enabled downloader middlewares:
  ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
   'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
   'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
   'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
   'scrapy.downloadermiddlewares.retry.RetryMiddleware',
   'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
   'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
   'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
   'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
   'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
   'scrapy.downloadermiddlewares.stats.DownloaderStats',
   'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware']
  2022-03-20 20:23:05 [scrapy.middleware] INFO: Enabled spider middlewares:
  ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
   'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
   'scrapy.spidermiddlewares.referer.RefererMiddleware',
   'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
   'scrapy.spidermiddlewares.depth.DepthMiddleware']
  2022-03-20 20:23:05 [scrapy.middleware] INFO: Enabled item pipelines:
  []
  2022-03-20 20:23:05 [scrapy.core.engine] INFO: Spider opened
  2022-03-20 20:23:05 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2022-03-20 20:23:05 [scrapy.extensions.httpcache] DEBUG: Using filesystem cache storage in /home/draco/docs/scraping/scrapyyy/upwork/.scrapy/httpcache
  2022-03-20 20:23:05 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024
  INITIAL REQUEST
  ---INFO: Requesting page=&gt; https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=COL&amp;linkID=tktldr&amp;shopperContext=&amp;caller=appList&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (405) &lt;GET https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=COL&amp;linkID=tktldr&amp;shopperContext=&amp;caller=appList&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=FOR&amp;linkID=tktldr&amp;shopperContext=&amp;caller=appList&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (405) &lt;GET https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=FOR&amp;linkID=tktldr&amp;shopperContext=&amp;caller=appList&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=AMP&amp;linkID=tktldr&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=AMP&amp;linkID=tktldr&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://budweisergardens.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONC&amp;linkID=global-labatt&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://budweisergardens.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONC&amp;linkID=global-labatt&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://pplcenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=global-allentown&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://pplcenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=global-allentown&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://ynottix.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EVENTS&amp;linkID=global-odu&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (405) &lt;GET https://ynottix.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EVENTS&amp;linkID=global-odu&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://csutickets.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=WC&amp;linkID=csuwc&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://csutickets.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=WC&amp;linkID=csuwc&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://tsongascenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=C&amp;linkID=global-lowell&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://tsongascenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=C&amp;linkID=global-lowell&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://wellsfargocenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=global-wachovia&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (405) &lt;GET https://wellsfargocenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=global-wachovia&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://stridebankcenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EGS&amp;linkID=global-enid&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (405) &lt;GET https://stridebankcenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EGS&amp;linkID=global-enid&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://cureinsurancearena.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CO&amp;linkID=global-sovereign&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://cureinsurancearena.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CO&amp;linkID=global-sovereign&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://ticketstar.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=RCCO&amp;linkID=pmi&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://ticketstar.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=RCCO&amp;linkID=pmi&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://hyveetix.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONC&amp;linkID=global-iowa&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://hyveetix.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONC&amp;linkID=global-iowa&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://portland5.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=pcpa&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://portland5.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=pcpa&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://selectyourtickets.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=PP&amp;linkID=rgp&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://selectyourtickets.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=PP&amp;linkID=rgp&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://ictickets.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=ICI&amp;linkID=nampa&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://ictickets.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=ICI&amp;linkID=nampa&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://umasstix.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=MCCON&amp;linkID=umass&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (200) &lt;GET https://umasstix.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=MCCON&amp;linkID=umass&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://xlcenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=XL&amp;linkID=global-hartford&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://xlcenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=XL&amp;linkID=global-hartford&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://tdplace.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EVENTS&amp;linkID=ottawa67&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (405) &lt;GET https://tdplace.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EVENTS&amp;linkID=ottawa67&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://liacourascenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=global-temple&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (405) &lt;GET https://liacourascenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=global-temple&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://libertyfirstcreditunionarena.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EVENTS.1&amp;linkID=global-ralston&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://libertyfirstcreditunionarena.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EVENTS.1&amp;linkID=global-ralston&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://semo.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=twsemo&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (405) &lt;GET https://semo.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=twsemo&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://treventscomplex.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CO&amp;linkID=global-bud&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://treventscomplex.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CO&amp;linkID=global-bud&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://xtreamarena.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=coralville-multi&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://xtreamarena.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=coralville-multi&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://ticketatlantic.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONCERT&amp;linkID=halifax&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (405) &lt;GET https://ticketatlantic.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONCERT&amp;linkID=halifax&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://enmaxcentre.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EC&amp;linkID=lethbridge-multi&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (403) &lt;GET https://enmaxcentre.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EC&amp;linkID=lethbridge-multi&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  ---INFO: Requesting page=&gt; https://ticketatlantic.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONCERT&amp;linkID=halifax&amp;shopperContext=&amp;caller=&amp;appCode=
  2022-03-20 20:23:05 [scrapy.core.engine] DEBUG: Crawled (405) &lt;GET https://ticketatlantic.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONCERT&amp;linkID=halifax&amp;shopperContext=&amp;caller=&amp;appCode=&gt; (referer: None) ['cached']
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;405 https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=COL&amp;linkID=tktldr&amp;shopperContext=&amp;caller=appList&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;405 https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=FOR&amp;linkID=tktldr&amp;shopperContext=&amp;caller=appList&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://ticketleader.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=AMP&amp;linkID=tktldr&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://budweisergardens.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONC&amp;linkID=global-labatt&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://pplcenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=global-allentown&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;405 https://ynottix.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EVENTS&amp;linkID=global-odu&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://csutickets.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=WC&amp;linkID=csuwc&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://tsongascenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=C&amp;linkID=global-lowell&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;405 https://wellsfargocenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=global-wachovia&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;405 https://stridebankcenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EGS&amp;linkID=global-enid&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://cureinsurancearena.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CO&amp;linkID=global-sovereign&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://ticketstar.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=RCCO&amp;linkID=pmi&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://hyveetix.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONC&amp;linkID=global-iowa&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://portland5.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=pcpa&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://selectyourtickets.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=PP&amp;linkID=rgp&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://ictickets.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=ICI&amp;linkID=nampa&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  ---INFO: General parser opened. PARSER1
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://xlcenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=XL&amp;linkID=global-hartford&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;405 https://tdplace.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EVENTS&amp;linkID=ottawa67&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;405 https://liacourascenter.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=global-temple&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://libertyfirstcreditunionarena.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EVENTS.1&amp;linkID=global-ralston&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;405 https://semo.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=twsemo&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://treventscomplex.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CO&amp;linkID=global-bud&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://xtreamarena.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CON&amp;linkID=coralville-multi&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;405 https://ticketatlantic.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONCERT&amp;linkID=halifax&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;403 https://enmaxcentre.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=EC&amp;linkID=lethbridge-multi&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:06 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;405 https://ticketatlantic.evenue.net/cgi-bin/ncommerce3/SEGetGroupList?groupCode=CONCERT&amp;linkID=halifax&amp;shopperContext=&amp;caller=&amp;appCode=&gt;: HTTP status code is not handled or not allowed
  2022-03-20 20:23:06 [scrapy.core.engine] INFO: Closing spider (finished)
  2022-03-20 20:23:06 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
  {'downloader/request_bytes': 15189,
   'downloader/request_count': 27,
   'downloader/request_method_count/GET': 27,
   'downloader/response_bytes': 304575,
   'downloader/response_count': 27,
   'downloader/response_status_count/200': 1,
   'downloader/response_status_count/403': 16,
   'downloader/response_status_count/405': 10,
   'elapsed_time_seconds': 0.444587,
   'finish_reason': 'finished',
   'finish_time': datetime.datetime(2022, 3, 20, 17, 23, 6, 67887),
   'httpcache/hit': 27,
   'httperror/response_ignored_count': 26,
   'httperror/response_ignored_status_count/403': 16,
   'httperror/response_ignored_status_count/405': 10,
   'log_count/DEBUG': 28,
   'log_count/INFO': 36,
   'memusage/max': 126562304,
   'memusage/startup': 126562304,
   'response_received_count': 27,
   'scheduler/dequeued': 27,
   'scheduler/dequeued/memory': 27,
   'scheduler/enqueued': 27,
   'scheduler/enqueued/memory': 27,
   'start_time': datetime.datetime(2022, 3, 20, 17, 23, 5, 623300)}
  2022-03-20 20:23:06 [scrapy.core.engine] INFO: Spider closed (finished)
  </code></pre>