title: git-p4 message and author encoding
body: <p>today i am in the position to migrate some pretty old perforce repositories to git. While this is realy interesting there is one thing that caught my eye. All special characters in the commit messages and even the author names are not in the correct encoding.</p>
  
  <p>So i tried to investigate where the problem comes from.</p>
  
  <ul>
  <li>first of all the perforce server does not support unicode, so setting the P4CHARSET has no effect but <code>Unicode clients require a unicode enabled server.</code></li>
  <li>then i checked the output of simple commands like <code>p4 users</code> wich where indeed in ANSI (consulting notepad++, or ISO-8859-1 according to <code>file -bi</code> on redirected output)</li>
  <li>the <code>locale</code> command says LANG=en_US.UTF-8 ...</li>
  </ul>
  
  <p>after all my guess is that all p4 client output is in ISO-8859-1 but git-p4 assumes UTF-8 instead.</p>
  
  <p>I tried rewriting the commit messages with</p>
  
  <pre><code>git filter-branch --msg-filter 'iconv -f iso-8859-1 -t utf-8' -- --all
  </code></pre>
  
  <p>but that doesnt fix the issues, especialy as it is not intended to rewrite the author names.</p>
  
  <p>anyone has a guess how to force the output to be translated to UTF-8 before git-p4 recieves them?</p>
  
  <p><strong>Update:</strong> </p>
  
  <p>i tried to "overwrite" the default p4 commands output with a simple shell script that i prepended to PATH</p>
  
  <pre><code>/usr/bin/p4 $@ | iconv -f iso-8859-1 -t utf-8
  </code></pre>
  
  <p>but that destoys the marshalled python objects that are obviously used:</p>
  
  <pre><code>  File "/usr/local/bin/git-p4", line 2467, in getBranchMapping
      for info in p4CmdList(command):
    File "/usr/local/bin/git-p4", line 480, in p4CmdList
      entry = marshal.load(p4.stdout)
  ValueError: bad marshal data
  </code></pre>
  
  <p><strong>Update2:</strong></p>
  
  <p>As seen here <a href="https://stackoverflow.com/questions/2276200/changing-default-encoding-of-python">Changing default encoding of Python?</a> i tried to set python encoding to ascii:</p>
  
  <pre><code>export export PYTHONIOENCODING="ascii"
  python -c 'import sys; print(sys.stdin.encoding, sys.stdout.encoding)'
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>('ascii', 'ascii')
  </code></pre>
  
  <p>but still all messages and authors are not correctly migrated.</p>
  
  <p><strong>Update 3:</strong></p>
  
  <p>Even trying to patch the git-p4.py <code>def commit(self, details, files, branch, parent = "")</code> function did not help:
  Changing</p>
  
  <pre><code>self.gitStream.write(details["desc"])
  </code></pre>
  
  <p>to one of those</p>
  
  <pre><code>self.gitStream.write(details["desc"].encode('utf8', 'replace'))
  self.gitStream.write(unicode(details["desc"],'utf8')
  </code></pre>
  
  <p>did just raise:</p>
  
  <pre><code>UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 29: ordinal not in range(128)
  </code></pre>
  
  <p>as i am no python developer i have no idea what to try next.</p>