title: YaCy unable to crawl & index intranet pages requiring user authentication
body: <p>I have just installed a local YaCy peer within the corporate intranet at my workplace to allow certain content to be crawled and indexed for searching. (That was the goal, at least.) However, most of the pages require users to authenticate themselves using their corporate credentials whenever they try to access them.</p>
  
  <p>This authentication requirement puts YaCy in a bad spot because it cannot access the pages it needs to process. It fails with the following error message (real internal hostname removed, but I can assure you it's valid):</p>
  
  <blockquote>
    <p>Crawling of "https://&lt; hostname-removed >" failed. Reason: scraper
    cannot load URL: java.io.IOException: REJECTED EMPTY RESPONSE BODY
    'HTTP/1.1 <strong>401 Authorization Required'</strong> for URL
    'https://&lt; hostname-removed >/'$/</p>
  </blockquote>
  
  <p>Even if someone's account needed to be used (not ideal, but ok), I've not (yet?) found a place where such credentials could be specified in the YaCy administration pages or config files.</p>
  
  <p>Is there a way YaCy can be made to crawl and index pages that require users to authenticate themselves?</p>
  
  <p>Thanks!</p>