title: Using wget to fake browser?
body: <p>I'd like to crawl a web site to build its sitemap.</p>
  
  <p>Problems is, the site uses an htaccess file to block spiders, so the following command only downloads the homepage (index.html) and stops, although it does contain links to other pages:</p>
  
  <pre><code>wget -mkEpnp -e robots=off -U Mozilla http://www.acme.com
  </code></pre>
  
  <p>Since I have no problem accessing the rest of the site with a browser, I assume the "-e robots=off -U Mozilla" options aren't enough to have wget pretend it's a browser.</p>
  
  <p>Are there other options I should know about? Does wget handle cookies by itself?</p>
  
  <p>Thank you.</p>
  
  <p>--</p>
  
  <p>Edit:
  I added those to wget.ini, to no avail:</p>
  
  <pre><code>hsts=0
  robots = off
  header = Accept-Language: en-us,en;q=0.5
  header = Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  header = Connection: keep-alive
  user_agent = Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0
  referer = /
  </code></pre>
  
  <p>--</p>
  
  <p>Edit: Found it.</p>
  
  <p>The pages linked to in the homepage were on a remote server, so wget would ignore them. Just add "--span-hosts" to tell wget to go there, and "-D www.remote.site.com" if you want to restrict spidering to that domain.</p>