title: C++ and UTF8 - Why not just replace ASCII?
body: <p>In my application I have to constantly convert string between <code>std::string</code> and <code>std::wstring</code> due different APIs (boost, win32, ffmpeg etc..). Especially with ffmpeg the strings end up utf8->utf16->utf8->utf16, just to open a file.</p>
  
  <p>Since UTF8 is backwards compatible with ASCII I thought that I consistently store all my strings UTF-8 <code>std::string</code> and only convert to <code>std::wstring</code> when I have to call certain unusual functions.</p>
  
  <p>This worked kind of well, I implemented to_lower, to_upper, iequals for utf8. However then I met several dead-ends std::regex, and regular string comparisons. To make this usable I would need to implement a custom <code>ustring</code> class based on std::string with re-implementation of all corresponding algorithms (including regex).</p>
  
  <p>Basically my conclusion is that utf8 is not very good for general usage. And the current <code>std::string/std::wstring</code> is mess.</p>
  
  <p>However, my question is why the default <code>std::string</code> and <code>""</code> are not simply changed to use UTF8? Especially as UTF8 is backward compatible? Is there possibly some compiler flag which can do this? Of course the stl implemention would need to be automatically adapted.</p>
  
  <p>I've looked at ICU, but it is not very compatible with apis assuming basic_string, e.g. no begin/end/c_str etc...</p>