title: Microsoft IDEs, source file encodings, BOMs and the Unicode character \uFEFF?
body: <p>We have parsers for various Microsoft languages (VB6, VB.net, C#, MS dialects of C/C++).</p>
  
  <p>They are Unicode enabled to the extent that we all agree on what Unicode is.  Where we don't agree, our lexers object.</p>
  
  <p>Recent MS IDEs all seem to read/write their source code files in UTF-8... I'm not sure this is always true.  Is there some reference document that makes it clear how MS will write a souce code file?  With or without byte order marks?  Does it vary from IDE version to version? (I can't imagine that the old VB6 dev environment wrote anything other than an 8 bit character set, and I'd guess it would be in the CP-xxxx encoding established by the locale, right?)</p>
  
  <p>For C# (and I assume other modern language dialects supported by MS), the character code \uFEFF can actually be found in the middle of a file.  This code is defined as a zero-width no-break space.    It appears to be ignored by VS 2010 when found in the middle of an identifier, in whitespace, but is significant in keywords and numbers.    So, what are the rules?   Or does MS have some kind of normalize-identifiers to handle things like composite characters, that allows different identifier strings to be treated as identical?</p>