title: Windows game: UTF-8, UTF-16, DirectX and Lua
body: <p>I'm developing a game for windows for learning purposes (I'm learning DirectX). I would like it to have UTF support.</p>
  
  <p><a href="https://stackoverflow.com/questions/402283/stdwstring-vs-stdstring">Reading this question</a> I learned that windows uses <code>wchar_t</code>, which is UTF-16. I want my game to have Lua scripting support, and <a href="http://lua-users.org/wiki/LuaUnicode" rel="nofollow noreferrer">Lua doesn't really like Unicode much</a>.. It simply treats strings as a "stream of bytes"; this works well enough for UTF-8, but UTF-16 would be virtually impossible to use.</p>
  
  <p>Long story short: windows wants UTF-16, lua wants UTF-8.</p>
  
  <p>So I thought, let's just use UTF-8 with normal <code>char*</code> and <code>string</code>! <code>.length()</code> will be messed up but who cares? However it doesn't work:</p>
  
  <pre><code>const char test_utf8[] = { 111, 108, 0xc3, 0xa9, 0 }; // UTF-8 for olè
  mFont-&gt;DrawTextA(0, test_utf8, -1, &amp;R, DT_NOCLIP, BLACK);
      /* DrawText is a Direct3d function to, well, draw text.
       * It's like MessageBox: it is a define to either DrawTextA
       * or DrawTextW, depending if unicode is defined or not. Here
       * we will use DrawTextA, since we are passing a normal char*. */
  </code></pre>
  
  <p>This prints <code>olÃ©</code>. In other words it doesn't appear to use UTF-8 but rather <a href="http://en.wikipedia.org/wiki/ISO/IEC_8859-1" rel="nofollow noreferrer">ISO-8859-1</a>.</p>
  
  <p>So, what can I do? I can think of the following:</p>
  
  <ol>
  <li>Abandon the idea of UTF; use ISO-8859-1 and be happy (this is what World of Warcraft does, at least for the enUS version)</li>
  <li>Convert every single string at every single frame from UTF-8 to UTF-16 (I'm worried about performance issues, though, considering it will do this 60+ times a second for each string and it's O(N) I'm pretty sure it will be fairly slow)</li>
  <li>For each lua string keep an UTF-16 copy; huge waste of memory, very difficult to implement (keeping the UTF-16 strings up to date when they change in Lua, etc)</li>
  </ol>