title: MITMProxy: smart URL replacement
body: <p>We use a custom scraper that have to take a separate website for a language (this is an architecture limitation). Like site1.co.uk, site1.es, site1.de etc.</p>
  
  <p>But we need to parse a website with many languages, separated by url - like site2.com/en, site2.com/de, site2.com/es and so on.</p>
  
  <p>I thought about MITMProxy: I could redirect all requests this way:</p>
  
  <pre><code>en.site2.com/* --&gt; site2.com/en
  de.site2.com/* --&gt; site2.com/de
  ...
  </code></pre>
  
  <p>I have written a small script which simply takes URLs and rewrites them:</p>
  
  <pre><code>class MyMaster(flow.FlowMaster):
  
    def handle_request(self, r):
      url = r.get_url()
  
      # replace URLs
      if 'blabla' in url:
        r.set_url(url.replace('something', 'another'))
  </code></pre>
  
  <p>But the target host generates 301 redirect with the response from the webserver - 'the page has been moved here' and the link to the site2.com/en</p>
  
  <p>It worked when I played with URL rewriting, i.e. site2.com/en --> site2.com/de.
  But for different hosts (subdomain and the root domain, to be precise), it does not work.</p>
  
  <p>I tried to replace the Host header in the handle_request method from above:</p>
  
  <pre><code>for key in r.headers.keys():
          if key.lower() == 'host':
              r.headers[key] = ['site2.com']
  </code></pre>
  
  <p>also I tried to replace the Referrer - all of that didn't help.</p>
  
  <p>How can I finally spoof that request from the subdomain to the main domain? If it generates a HTTP(s) client warning it's ok since we need that for the scraper (and the warnings there can be turned off), not the real browser.</p>
  
  <p>Thanks!</p>