title: Why am I getting 400 Bad Requests with these proxy servers?
body: <p>So I'm pretty new to networking and the use of proxy servers. I have a scraper that scrapes certain websites, but I realize that I need to alter my IP addresses and whatnot so that I don't get booted from the website. I found the following program on GitHub that I want to use:</p>
  
  <p><a href="https://github.com/aivarsk/scrapy-proxies" rel="nofollow">https://github.com/aivarsk/scrapy-proxies</a></p>
  
  <p>I have everything implemented as follows:</p>
  
  <p>Spider:</p>
  
  <pre><code># This package will contain the spiders of your Scrapy project
  #
  # Please refer to the documentation for information on how to create and manage
  # your spiders.
  import scrapy
  from scrapy.spiders import CrawlSpider, Rule
  from scrapy.linkextractors import LinkExtractor
  from backpage_scrape import items
  #from toolz import first
  #import ipdb
  #from lxml import html
  from datetime import datetime, timedelta
  import os
  
  HOME = os.environ['HOMEPATH']
  os.chdir(HOME +       "/Desktop/GitHub/Rover/backpage_scrape/backpage_scrape/spiders/") 
  
  # Method that gets today's date
  def backpage_date_today():
      now = datetime.utcnow() - timedelta(hours=4)
      weekdays = ['Mon. ','Tue. ','Wed. ','Thu. ','Fri. ','Sat. ','Sun. ']
      months = ['Jan. ','Feb. ','Mar. ','Apr. ','May. ', 'Jun. ','Jul. ','Aug. ','Sep. ','Oct. ','Nov. ','Dec. ']
      backpage_date = weekdays[now.weekday()] + months[now.month-1] + str(now.day)
      return backpage_date
  
  # Method that gets yesterday's date
  def backpage_date_yesterday():
      now = datetime.utcnow() - timedelta(days=1, hours=4)
      weekdays = ['Mon. ','Tue. ','Wed. ','Thu. ','Fri. ','Sat. ','Sun. ']
  months = ['Jan. ','Feb. ','Mar. ','Apr. ','May. ', 'Jun. ','Jul. ','Aug. ','Sep. ','Oct. ','Nov. ','Dec. ']
  backpage_date = weekdays[now.weekday()] + months[now.month-1] + str(now.day)
  return backpage_date
  
  # Open file which contains input urls
  with open("test_urls.txt","rU") as infile:
      urls = [row.strip("\n") for row in infile]
  
  class BackpageSpider(CrawlSpider):
  name = 'backpage'
  allowed_domains = ['backpage.com']
  start_urls = urls
  
  def parse(self,response):
  
      if response.status &lt; 600:
  
          todays_links = []
  
          backpage_date = backpage_date_today()
          yesterday_date = backpage_date_yesterday()
  
          if backpage_date in response.body:
              # Get all URLs to iterate through
              todays_links = response.xpath("//div[@class='date'][1]/following-sibling::div[@class='date'][1]/preceding-sibling::div[preceding-sibling::div[@class='date']][contains(@class, 'cat')]/a/@href").extract()
  
          # timeOut = 0
          for url in todays_links: 
              # Iterate through pages and scrape
              # if timeOut == 10:
              #   time.sleep(600)
              #   timeOut = 0
              # else:
              #   timeOut += 1
  
              yield scrapy.Request(url,callback=self.parse_ad_into_content)
  
          for url in set(response.xpath('//a[@class="pagination next"]/@href').extract()):
              yield scrapy.Request(url,callback=self.parse)
  
      else:
          time.sleep(600)
          yield scrapy.Request(response.url,callback=self.parse)
  
  # Parse page
  def parse_ad_into_content(self,response):
      item = items.BackpageScrapeItem(url=response.url,
          backpage_id=response.url.split('.')[0].split('/')[2].encode('utf-8'),
          text = response.body,
          posting_body= response.xpath("//div[@class='postingBody']").extract()[0].encode('utf-8'),
          date = datetime.utcnow()-timedelta(hours=5),
          posted_date = response.xpath("//div[@class='adInfo']/text()").extract()[0].encode('utf-8'),
          posted_age = response.xpath("//p[@class='metaInfoDisplay']/text()").extract()[0].encode('utf-8'),
          posted_title = response.xpath("//div[@id='postingTitle']//h1/text()").extract()[0].encode('utf-8')
          )
      return item
  </code></pre>
  
  <p>Portion of settings.py:</p>
  
  <pre><code># Retry many times since proxies often fail
  RETRY_TIMES = 10
  # Retry on most error codes since proxies fail for different reasons
  RETRY_HTTP_CODES = [500, 503, 504, 400, 403, 404, 408]
  
  DOWNLOADER_MIDDLEWARES = {
      'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
      # Fix path to this module
      'backpage_scrape.randomproxy.RandomProxy': 100,
      'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
  }
  
  PROXY_LIST = 'C:/Users/LPrice/Desktop/GitHub/Rover/backpage_scrape/backpage_scrape/proxies.txt'
  </code></pre>
  
  <p>randomproxy.py is the exact same as it is on the GitHub link.</p>
  
  <p>Proxies.txt:</p>
  
  <pre><code>https://6.hidemyass.com/ip-4
  https://5.hidemyass.com/ip-1
  https://4.hidemyass.com/ip-1
  https://4.hidemyass.com/ip-2
  https://4.hidemyass.com/ip-3
  https://3.hidemyass.com/ip-1
  https://3.hidemyass.com/ip-2
  https://3.hidemyass.com/ip-3
  https://2.hidemyass.com/ip-1
  https://2.hidemyass.com/ip-2
  https://2.hidemyass.com/ip-3
  https://1.hidemyass.com/ip-1
  https://1.hidemyass.com/ip-2
  https://1.hidemyass.com/ip-3
  https://1.hidemyass.com/ip-4
  https://1.hidemyass.com/ip-5
  https://1.hidemyass.com/ip-6
  https://1.hidemyass.com/ip-7
  https://1.hidemyass.com/ip-8
  </code></pre>
  
  <p>So if you look at the top of the GitHub README, you'll see that it says to "copy-paste into text file and reformat to <a href="http://host:port" rel="nofollow">http://host:port</a> format." I'm not sure how I do that, or if already is in that format.</p>
  
  <p>Like I said, my errors are 400 Bad Requests. I'm not sure if it's useful, but the console is saying:</p>
  
  <pre><code>Retrying &lt;GET http://sf.backpage.com/restOfURL&gt; &lt;failed 10 times&gt;: 400 Bad Request
  </code></pre>
  
  <p>Is it supposed to be showing the proxies in the URL above, before the "sf.backpage.com" portion? </p>
  
  <p>Thanks a ton for your time...I really appreciate the help.</p>
  
  <p>EDIT: Also, I wasn't sure where/how to insert the snippet of code at the bottom of the GitHub's README. Any suggestions on that would also be useful.</p>