title: Scrapy ignoring content after # tag in the URL
body: <p>Hi i am scraping a site which has the URl like below</p>
  
  <pre><code>http://www.example.com/categories-Mobile-Phones.aspx#RSS=pgZZ1QQdivZZctl00_ContentPlaceHolder1_ctl00_ctl03
  </code></pre>
  
  <p>i had placed this in <code>start_url</code> and requested a response , but i received the response as below</p>
  
  <pre><code>&lt;200 http://www.example.com/categories-Mobile-Phones.aspx&gt;
  </code></pre>
  
  <p>That is, it simply ignores the content after <code>hash tag</code>, so i had read some post and came to know that, when we request a url with hash tag the server simply ignores the hash fragments, i mean hash fragments are used to load some extra information for ajax or javascript requests.so i had replaced the url in <code>start_url</code> with an <code>exclamatory mark(!)</code> after <code># tag</code> as below</p>
  
  <pre><code>http://www.example.com/categories-Mobile-Phones.aspx#!RSS=pgZZ1QQdivZZctl00_ContentPlaceHolder1_ctl00_ctl03
  </code></pre>
  
  <p>And now the output is </p>
  
  <pre><code>&lt;GET http://www.example.com/categories-Mobile-Phones.aspx?_escaped_fragment_=RSS=pgZZ1QQdivZZctl00_ContentPlaceHolder1_ctl00_ctl03
  </code></pre>
  
  <p>I had done this by reading this post <code>https://developers.google.com/webmasters/ajax-crawling/docs/getting-started</code>,  According to the concept in this post i need to convert the output URL with <code>?_escaped_fragment_=</code> to the URL containing <code>#</code> (I mean the original URL)to parse the page completely without scrapy ignoring the hash fragment.How to convert it.</p>
  
  <p>I hope i explained well, if not please correct me and let me know the concept of how to make scrapy of not ignoring the hash fragments from a URL.</p>
  
  <p>Thanks in advance............... </p>