title: PHP: removing invalid utf-8 characters in XML using filter
body: <p>I have a large file, so I have created a filter for removing invalid utf-8 characters from XML.</p>
  <pre><code>class ValidUTF8XMLFilter extends php_user_filter {
  
      protected static $pattern = '/([\x09\x0A\x0D\x20-\x7E]|[\xC2-\xDF][\x80-\xBF]|\xE0[\xA0-\xBF][\x80-\xBF]|[\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}|\xED[\x80-\x9F][\x80-\xBF]|\xF0[\x90-\xBF][\x80-\xBF]{2}|[\xF1-\xF3][\x80-\xBF]{3}|\xF4[\x80-\x8F][\x80-\xBF]{2})|./x';
  
      function filter($in, $out, &amp;$consumed, $closing)
      {
          while ($bucket = stream_bucket_make_writeable($in)) {
              $bucket-&gt;data = preg_replace(self::$pattern, '$1', $bucket-&gt;data);
              $consumed += $bucket-&gt;datalen;
              stream_bucket_append($out, $bucket);
          }
          return PSFS_PASS_ON;
      }
  }
  </code></pre>
  <p>This filter will remove also utf-8 characters not only invalid in xml, but also in utf-8.
  The regex is taken from <a href="https://www.w3.org/International/questions/qa-forms-utf-8.en" rel="nofollow noreferrer">Multilingual form encoding</a>. The class was taken from this answer: <a href="https://stackoverflow.com/questions/3466035/how-to-skip-invalid-characters-in-xml-file-using-php/3466609#3466609">How to skip invalid characters in XML file using PHP</a> and rewritten. The pattern in that answer won't work for invalid utf-8 characters, eg. 0x1D.</p>
  <p>Will this filter work, in situation, where invalid bytes starts at the end of buffer and ends in beginning of next filtering? Is this situation possible?</p>