title: Unit-testing functions written in Scrapy
body: <p>I recently started a simple project that requires scraping and picked up <strong>Scrapy</strong> for the job.</p>
  <p>I have a number of <strong>functions</strong> in my spider and I want to <strong>test them independently</strong>.</p>
  <p><strong>Is there a framework I can use?</strong></p>
  <ul>
  <li><p>I found a similar question <a href="https://stackoverflow.com/questions/6456304/scrapy-unit-testing">here</a>, but the answers aren't clear or satisfactory to me. The question is also pretty old.</p>
  </li>
  <li><p>I am looking for <strong>something similar to <em>JUnit</em> in <em>Java</em></strong>.</p>
  </li>
  <li><p>The <strong>Scrapy documentation</strong> mentions something about <strong>Twisted</strong>, but I <strong>couldn't find much info on that</strong>.</p>
  </li>
  <li><p><strong>Spider contracts aren't what I need.</strong> I would like to test each function independently.</p>
  </li>
  </ul>
  <p>Here is an example of a function I got:</p>
  <pre class="lang-py prettyprint-override"><code>def get_urls_range(today, date_start, date_end):
      &quot;&quot;&quot;Returns all nachrichtenarchiv URLs per date for a given timerange.
  
      Args:
          today:
              Boolean, if True include today in the range.
          date_start:
              Object of type datetime to represent the earliest date from which URLs are to be
              generated.
          date_end object:
              Object of type datetime to represent the latest date from which URLs are to be
              generated.
  
      Returns:
          A list of URLs. Each URL leads to a page that contains links to all articles that have been
          posted on a certain date.
      &quot;&quot;&quot;
  
      # By default date_end is not included in the range as it doesn't contain all the articles from
      # date.today() and it is expexted that the function is called for a range between some start
      # date and today. This is why start_urls is by default initiated with the 'today' liink that
      # doesn't follow the same naming convention as the archive. This if-else allows flexibility with
      # date_end.
      if today :
          start_urls = [&quot;https://www.spiegel.de/nachrichtenarchiv/&quot;]
      else:
          start_urls = []
          date_end = date_end + timedelta(1)
  
      # Go through all dates and add their respective urls to start_urls
      for num_days in range(int((date_end - date_start).days)):
          url_to_append = (date_start + timedelta(num_days)).strftime(&quot;https://www.spiegel.de/nachrichtenarchiv/artikel-%d.%m.%Y.html&quot;)
          start_urls.insert(1, url_to_append)
      return start_urls
  </code></pre>
  <p>And here is another one:</p>
  <pre class="lang-py prettyprint-override"><code>def get_article_text(response):
      &quot;&quot;&quot;Returns an article's text.
  
      Args:
          response:
              Response object cotaining an article page.
  
      Reeturns:
          The article's text in a String.
  
      Raises:
          Exception: Article text could not be scraped.
      &quot;&quot;&quot;
  
      try:
          article_text_intro = response.xpath('normalize-space(//header[@data-area=&quot;intro&quot;]//div[@class=&quot;RichText RichText--sans leading-loose lg:text-xl md:text-xl sm:text-l lg:mb-32 md:mb-32 sm:mb-24&quot;]/text())').get()
          article_text_list = ''.join([text.strip() for text in response.xpath('//div[@data-area=&quot;body&quot;]/div[@data-sara-click-el=&quot;body_element&quot; and @data-area=&quot;text&quot;]//p//text()').getall()])
          article_text = article_text_intro + article_text_list
  
          return article_text
      except Exception:
          print(&quot;A problem occurred while scraping the article's text.&quot;)
          return &quot;N/A&quot;
  </code></pre>
  <p>Do you have any suggestions?</p>
  <p>Are there approaches towards unit-testing that I am missing entirely?</p>