title: How to add Proxy to Scrapy and Selenium Script
body: <p>I would like to add a proxy to my script.</p>
  
  <p>How do I have to do it? Do I have to use Selenium or Scrapy for it?</p>
  
  <p>I think that Scrapy is making the initial request, so it would make sense to use scrapy for it. But what exactly do I have to do?</p>
  
  <p>Can you recommend any proxylist which works quite reliable?</p>
  
  <p>This is my current script:</p>
  
  <pre><code># -*- coding: utf-8 -*-
  import scrapy
  from scrapy import Spider
  from scrapy.selector import Selector
  from scrapy.http import Request
  
  from selenium.common.exceptions import NoSuchElementException
  from selenium.webdriver.common.keys import Keys
  from selenium.webdriver.common.action_chains import ActionChains
  from selenium import webdriver
  
  import re
  import csv
  from time import sleep
  
  class PostsSpider(Spider):
      name = 'posts'
      allowed_domains = ['xyz']
      start_urls = ('xyz',)
  
      def parse(self, response):
          with open("urls.txt", "rt") as f:
              start_urls = [url.strip() for url in f.readlines()]
              for url in start_urls:
                  self.driver = webdriver.Chrome('C:\webdrivers\chromedriver.exe')
                  self.driver.get(url)
  
                  try:
                      self.driver.find_element_by_id('currentTab').click()
                      sleep(3)
                      self.logger.info('Sleeping for 5 sec.')
                      self.driver.find_element_by_xpath('//*[@id="_blog-menu"]/div[2]/div/div[2]/a[3]').click()
                      sleep(7)
                      self.logger.info('Sleeping for 7 sec.')
                  except NoSuchElementException:
                      self.logger.info('Blog does not exist anymore')
  
                  while True:                 
                      try:
                          element = self.driver.find_element_by_id('last_item')
                          self.driver.execute_script("arguments[0].scrollIntoView(0, document.documentElement.scrollHeight-5);", element)
                          sleep(3)
                          self.driver.find_element_by_id('last_item').click()
                          sleep(7)
  
  
                      except NoSuchElementException:
                          self.logger.info('No more tipps')                       
                          sel = Selector(text=self.driver.page_source)
                          allposts = sel.xpath('//*[@class="block media _feedPick feed-pick"]')
  
                          for post in allposts:
                              username = post.xpath('.//div[@class="col-sm-7 col-lg-6 no-padding"]/a/@title').extract()
                              publish_date = post.xpath('.//*[@class="bet-age text-muted"]/text()').extract()
  
                              yield {'Username': username,
                                  'Publish date': publish_date}
                          self.driver.close()
                          break
  </code></pre>