title: Fetching only X/HTML links (not images) based on mime type
body: <p>I'm crawling a site using Ruby + OpenURI + Nokogiri. Fetch a page, find all the <code>a[href]</code> and (if they're in the same domain and right protocol) follow them to crawl again.</p>
  
  <p>Sometimes there are links to large binaries (e.g. jpeg, exe), and I don't want to crawl those.</p>
  
  <p>I tried using the <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html" rel="nofollow">HTTP "Accept" header</a> to get an error or empty response for the wrong mime types like so:</p>
  
  <pre><code>require 'open-uri'
  page = open(url, 'Accept'=&gt;'text/html,application/xhtml+xml,application/xml')
  </code></pre>
  
  <p>...but OpenURI still downloads binaries sent with another mime type.</p>
  
  <p>Other than looking at file extensions in the url for a <em>probable</em> file type, <strong>how can I prevent the download (or detect a conflicting response type) for an arbitrary URL</strong>?</p>