title: Spynner wrong encoding
body: <p>I'm trying to download this page - <a href="https://itunes.apple.com/ru/app/farm-story/id367107953?mt=8" rel="nofollow">https://itunes.apple.com/ru/app/farm-story/id367107953?mt=8</a> (looks like this for me in Russia - <a href="http://screencloud.net/v/6a7o" rel="nofollow">http://screencloud.net/v/6a7o</a>) via <a href="https://pypi.python.org/pypi/spynner#intro" rel="nofollow">spynner</a> in python - it uses some javascript checking so one does not simply download it without full browser emulation.</p>
  
  <p>My code:</p>
  
  <pre><code># -*- coding: utf-8 -*-
  import sys
  reload(sys)
  sys.setdefaultencoding('utf-8')
  
  from StringIO import StringIO
  import spynner
  
  
  def log(str, filename_end):
      filename = '/tmp/apple_log_%s.html' % filename_end
      print 'logged to %s' % filename
      f = open(filename, 'w')
      f.write(str)
      f.close()
  
  debug_stream = StringIO()
  browser = spynner.Browser(debug_level=3, debug_stream=debug_stream)
  
  browser.load("https://itunes.apple.com/ru/app/farm-story/id367107953?mt=8")
  
  ret = browser.contents
  log(ret, 'noenc')
  
  print 'content length = %s' % len(ret)
  browser.close()
  del browser
  
  f=open('/tmp/apple_log_debug', 'w')
  f.write(debug_stream.getvalue())
  f.close()
  print 'log stored in /tmp/debug_log'
  </code></pre>
  
  <p>So, the problem is: either apple, either spynner work wrong with Cyrillic symbols. I see them fine if I try <code>browser.show()</code> after loading, but in the code and logs they are still wrong encoded like <code>&lt;meta content="ÐÐ¾Ð»ÑÑÐ¸ÑÑ Farm Storyâ¢ Ð² App Store. ÐÑÐ¾ÑÐ¼Ð¾ÑÑÐµÑÑ ÑÐºÑÐ¸Ð½ÑÐ¾ÑÑ Ð¸ ÑÐµÐ¹ÑÐ¸Ð½Ð³Ð¸, Ð¿ÑÐ¾ÑÐ¸ÑÐ°ÑÑ Ð¾ÑÐ·ÑÐ²Ñ Ð¿Ð¾ÐºÑÐ¿Ð°ÑÐµÐ»ÐµÐ¹." property="og:description"&gt;</code>.</p>
  
  <p><a href="http://2cyr.com/" rel="nofollow">http://2cyr.com/</a> Says that it is a <code>utf-8</code> text displayed like <code>iso-8859-1</code>...</p>
  
  <p>As you see - I don't use any headers in my request, but if I take them from chrome's network debug console and pass it to <code>load()</code> method e.g. <code>headers=[('Accept-Encoding', 'utf-8'), ('Accept-Language', 'ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4')]</code> - I get the same result.</p>
  
  <p>Also, from the same network console you can see that chrome uses <code>gzip,deflate,sdch</code> as Accept-Encoding. I can try that too, but I fail to decode what I get: <code>&lt;html&gt;&lt;head&gt;&lt;/head&gt;&lt;body&gt;ï¿½ï¿½}ksÇï¿½g!ï¿½ï¿½ï¿½4ï¿½I/zï¿½Oï¿½ï¿½ï¿½/)ï¿½(ywï¿½ï¿½ï¿½é®iï¿½ï¿½{ï¿½&lt;vï¿½ï¿½ï¿½:ï¿½ï¿½Ù·ï¿½Ø³-?ï¿½bï¿½bï¿½ï¿½ jï¿½...</code> even if I remove the tags from the begin and end of the result.</p>
  
  <p>Any help?</p>