title: ASP.NET - Unable to translate Unicode character XXX at index YYY to specified code page
body: <p>On a ASP.NET 4 website and im getting the following error when trying to load data from the database into a GridView.</p>
  
  <p><strong>Unable to translate Unicode character \uD83D at index 49 to specified code page.</strong></p>
  
  <p>I've found out that this happens when a data row contains:
  <strong>Text Text Text </strong></p>
  
  <p>As i understand this text cannot be translated into a valid utf-8 response.</p>
  
  <ol>
  <li><p>Is that really the reason?</p></li>
  <li><p>Is there a way to clean the text before loading it into the gridview to prevent such errors?</p></li>
  </ol>
  
  <hr>
  
  <p><strong>UPDATE:</strong></p>
  
  <p>I have some progress
  I've found out I only get this error when I'm using Substring method on a string. 
  (I'm using substring to show part of the text as a preview to the user).</p>
  
  <p><strong>For example in an ASP.NET Web Form I do this:</strong></p>
  
  <pre><code>String txt = test ;
  
  //txt string can also be created by 
  String txt = char.ConvertFromUtf32(116) + char.ConvertFromUtf32(101) +char.ConvertFromUtf32(115) + char.ConvertFromUtf32(116) + char.ConvertFromUtf32(32) + char.ConvertFromUtf32(128148);
  
  // this works ok txt is shown in the webform label.
  Label1.Text = txt; 
  
  //length is equal to 7.
  Label2.Text = txt.Length.ToString();
  
  //causes exception - Unable to translate Unicode character \uD83D at index 5 to specified code page.
  Label3.Text = txt.Substring(0, 6);
  </code></pre>
  
  <p>I know that .NET string is based on utf-16 which supports surrogate pairs.</p>
  
  <p>When i'm using SubString function I accidently break the surrogate pair and causes the exception.
      I found out that I can use <strong>StringInfo class</strong>:</p>
  
  <pre><code>var si = new System.Globalization.StringInfo(txt);
  var l = si.LengthInTextElements; // length is equal to 6.
  Label3.Text = si.SubstringByTextElements(0, 5); //no exception!
  </code></pre>
  
  <p><strong>Another alternative is to just delete the surrogate pairs :</strong></p>
  
  <pre><code>Label3.Text = ValidateUtf8(txt).Substring(0, 3); //no exception!
  
      public static string ValidateUtf8(string txt)
              {
                  StringBuilder sbOutput = new StringBuilder();
                  char ch;
  
                  for (int i = 0; i &lt; body.Length; i++)
                  {
                      ch = body[i];
                      if ((ch &gt;= 0x0020 &amp;&amp; ch &lt;= 0xD7FF) ||
                              (ch &gt;= 0xE000 &amp;&amp; ch &lt;= 0xFFFD) ||
                              ch == 0x0009 ||
                              ch == 0x000A ||
                              ch == 0x000D)
                      {
                          sbOutput.Append(ch);
                      }
  
                  }
                  return sbOutput.ToString();
              }
  </code></pre>
  
  <p>Is this really a problem of surrogate pairs?</p>
  
  <p>Which characters use surrogate pairs ? is there a list?</p>
  
  <p>Should I keep support for surrogate pairs? should i go with using StringInfo Class or just delete non valid chars?</p>
  
  <p>Thanks!</p>