title: How to take care of badly formed URLs, such as those containing "&#xA" and whitespaces while using recursive wget
body: <p>I am trying to crawl a mid-sized (a few thousand pages) domain using recursive wget. I am using the following command (on Ubuntu linux 13.10): </p>
  
  <pre><code>wget -r --random-wait -D example.com http://www.example.com
  </code></pre>
  
  <p>A few pages go through fine, but after that wget tries to download a series of pages for which the URL is malformed. When I view the source of some page, I see relative links such as:</p>
  
  <pre><code>&lt;a href="&amp;#xA;                    displayPage.aspx?arg1=val1&amp;#xA;                  "&gt;Another page&lt;/a&gt;
  </code></pre>
  
  <p>These are relative links, and wget is trying to download the page: </p>
  
  <pre><code>http://www.example.com/subdirectory/%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20displayPage.aspx?arg1=val1
  </code></pre>
  
  <p>and gets </p>
  
  <pre><code>HTTP request sent, awaiting response... 400 Bad Request
  </code></pre>
  
  <p>On the browser these characters are not shown, when I take the mouse to such a link, the link reads nicely as </p>
  
  <pre><code>http://www.example.com/subdirectory/displayPage.aspx?arg1=val1
  </code></pre>
  
  <p>and so the website functions perfectly when I view it manually. Is there any way to make recursive wget work fine with a domain having such links? I could not find any post which discussed this, by searching in Google or directly here.</p>