title: Converting accented characters and HTML entities into UTF-8?
body: <p>I'm working on a project that will allow me to download stories from Portkey.org for reading on my kindle, and I can't for the life of me figure out how to properly encode/parse the grabbed HTML from the website. I am using <a href="http://sourceforge.net/projects/simplehtmldom/" rel="nofollow">simple_html_dom</a> to grab it, and am passing the <code>innertext</code> of the main element where the story is held for parsing.</p>
  
  <p>So what I'm trying to accomplish here is the following:</p>
  
  <ol>
  <li>Grab HTML from Portkey.org story</li>
  <li>Convert all HTML Entities on page to regular characters for reading (entities like <code>&amp;rdquo;</code> to <code>”</code>, <code>&amp;ldquo;</code> to <code>“</code>, <code>&amp;hellip;</code> to <code>…</code> and so on)</li>
  <li>Any accented characters or characters of other languages (like Korean, Japanese, Chinese, etc.) should stay as they are.</li>
  <li>Fix the HTML using <code>tidy</code> and save it to a <code>.html</code> file.</li>
  </ol>
  
  <p>Everything I have tried so far results in either of the following:</p>
  
  <ul>
  <li>Diamond with question mark inside of it where the accented characters should be</li>
  <li>Broken UTF-8 characters where there should be quotations and ellipses, but accented characters show correctly</li>
  </ul>
  
  <p>A sample from the story HTML:</p>
  
  <pre><code>&lt;p&gt; Wel [snip] your emotions&amp;hellip;but most impor [snip] ng fiancé &lt;/p&gt;
  </code></pre>
  
  <p><strong>EDIT</strong></p>
  
  <p><code>html_entity_decode</code> results in the following output:</p>
  
  <pre><code> Wel [snip] your emotionsâ€¦but most impor [snip] ng fiancé
  </code></pre>
  
  <p>As you can see, the accented character is correct, but the <code>&amp;hellip;</code> now displays incorrectly.</p>
  
  <p><strong>EDIT 2:</strong></p>
  
  <p>Results of <code>get_html_translation_table(HTML_ENTITIES)</code>:</p>
  
  <pre><code>array(252) { ["""]=&gt; string(6) """ ["&amp;"]=&gt; string(5) "&amp;" ["&lt;"]=&gt; string(4) "&lt;" ["&gt;"]=&gt; string(4) "&gt;" ["Â "]=&gt; string(6) " " ["Â¡"]=&gt; string(7) "¡" ["Â¢"]=&gt; string(6) "¢" ["Â£"]=&gt; string(7) "£" ["Â¤"]=&gt; string(8) "¤" ["Â¥"]=&gt; string(5) "¥" ["Â¦"]=&gt; string(8) "¦" ["Â§"]=&gt; string(6) "§" ["Â¨"]=&gt; string(5) "¨" ["Â©"]=&gt; string(6) "©" ["Âª"]=&gt; string(6) "ª" ["Â«"]=&gt; string(7) "«" ["Â¬"]=&gt; string(5) "¬" ["Â­"]=&gt; string(5) "­" ["Â®"]=&gt; string(5) "®" ["Â¯"]=&gt; string(6) "¯" ["Â°"]=&gt; string(5) "°" ["Â±"]=&gt; string(8) "±" ["Â²"]=&gt; string(6) "²" ["Â³"]=&gt; string(6) "³" ["Â´"]=&gt; string(7) "´" ["Âµ"]=&gt; string(7) "µ" ["Â¶"]=&gt; string(6) "¶" ["Â·"]=&gt; string(8) "·" ["Â¸"]=&gt; string(7) "¸" ["Â¹"]=&gt; string(6) "¹" ["Âº"]=&gt; string(6) "º" ["Â»"]=&gt; string(7) "»" ["Â¼"]=&gt; string(8) "¼" ["Â½"]=&gt; string(8) "½" ["Â¾"]=&gt; string(8) "¾" ["Â¿"]=&gt; string(8) "¿" ["Ã€"]=&gt; string(8) "À" ["Ã"]=&gt; string(8) "Á" ["Ã‚"]=&gt; string(7) "Â" ["Ãƒ"]=&gt; string(8) "Ã" ["Ã„"]=&gt; string(6) "Ä" ["Ã…"]=&gt; string(7) "Å" ["Ã†"]=&gt; string(7) "Æ" ["Ã‡"]=&gt; string(8) "Ç" ["Ãˆ"]=&gt; string(8) "È" ["Ã‰"]=&gt; string(8) "É" ["ÃŠ"]=&gt; string(7) "Ê" ["Ã‹"]=&gt; string(6) "Ë" ["ÃŒ"]=&gt; string(8) "Ì" ["Ã"]=&gt; string(8) "Í" ["ÃŽ"]=&gt; string(7) "Î" ["Ã"]=&gt; string(6) "Ï" ["Ã"]=&gt; string(5) "Ð" ["Ã‘"]=&gt; string(8) "Ñ" ["Ã’"]=&gt; string(8) "Ò" ["Ã“"]=&gt; string(8) "Ó" ["Ã”"]=&gt; string(7) "Ô" ["Ã•"]=&gt; string(8) "Õ" ["Ã–"]=&gt; string(6) "Ö" ["Ã—"]=&gt; string(7) "×" ["Ã˜"]=&gt; string(8) "Ø" ["Ã™"]=&gt; string(8) "Ù" ["Ãš"]=&gt; string(8) "Ú" ["Ã›"]=&gt; string(7) "Û" ["Ãœ"]=&gt; string(6) "Ü" ["Ã"]=&gt; string(8) "Ý" ["Ãž"]=&gt; string(7) "Þ" ["ÃŸ"]=&gt; string(7) "ß" ["Ã "]=&gt; string(8) "à" ["Ã¡"]=&gt; string(8) "á" ["Ã¢"]=&gt; string(7) "â" ["Ã£"]=&gt; string(8) "ã" ["Ã¤"]=&gt; string(6) "ä" ["Ã¥"]=&gt; string(7) "å" ["Ã¦"]=&gt; string(7) "æ" ["Ã§"]=&gt; string(8) "ç" ["Ã¨"]=&gt; string(8) "è" ["Ã©"]=&gt; string(8) "é" ["Ãª"]=&gt; string(7) "ê" ["Ã«"]=&gt; string(6) "ë" ["Ã¬"]=&gt; string(8) "ì" ["Ã­"]=&gt; string(8) "í" ["Ã®"]=&gt; string(7) "î" ["Ã¯"]=&gt; string(6) "ï" ["Ã°"]=&gt; string(5) "ð" ["Ã±"]=&gt; string(8) "ñ" ["Ã²"]=&gt; string(8) "ò" ["Ã³"]=&gt; string(8) "ó" ["Ã´"]=&gt; string(7) "ô" ["Ãµ"]=&gt; string(8) "õ" ["Ã¶"]=&gt; string(6) "ö" ["Ã·"]=&gt; string(8) "÷" ["Ã¸"]=&gt; string(8) "ø" ["Ã¹"]=&gt; string(8) "ù" ["Ãº"]=&gt; string(8) "ú" ["Ã»"]=&gt; string(7) "û" ["Ã¼"]=&gt; string(6) "ü" ["Ã½"]=&gt; string(8) "ý" ["Ã¾"]=&gt; string(7) "þ" ["Ã¿"]=&gt; string(6) "ÿ" ["Å’"]=&gt; string(7) "Œ" ["Å“"]=&gt; string(7) "œ" ["Å "]=&gt; string(8) "Š" ["Å¡"]=&gt; string(8) "š" ["Å¸"]=&gt; string(6) "Ÿ" ["Æ’"]=&gt; string(6) "ƒ" ["Ë†"]=&gt; string(6) "ˆ" ["Ëœ"]=&gt; string(7) "˜" ["Î‘"]=&gt; string(7) "Α" ["Î’"]=&gt; string(6) "Β" ["Î“"]=&gt; string(7) "Γ" ["Î”"]=&gt; string(7) "Δ" ["Î•"]=&gt; string(9) "Ε" ["Î–"]=&gt; string(6) "Ζ" ["Î—"]=&gt; string(5) "Η" ["Î˜"]=&gt; string(7) "Θ" ["Î™"]=&gt; string(6) "Ι" ["Îš"]=&gt; string(7) "Κ" ["Î›"]=&gt; string(8) "Λ" ["Îœ"]=&gt; string(4) "Μ" ["Î"]=&gt; string(4) "Ν" ["Îž"]=&gt; string(4) "Ξ" ["ÎŸ"]=&gt; string(9) "Ο" ["Î "]=&gt; string(4) "Π" ["Î¡"]=&gt; string(5) "Ρ" ["Î£"]=&gt; string(7) "Σ" ["Î¤"]=&gt; string(5) "Τ" ["Î¥"]=&gt; string(9) "Υ" ["Î¦"]=&gt; string(5) "Φ" ["Î§"]=&gt; string(5) "Χ" ["Î¨"]=&gt; string(5) "Ψ" ["Î©"]=&gt; string(7) "Ω" ["Î±"]=&gt; string(7) "α" ["Î²"]=&gt; string(6) "β" ["Î³"]=&gt; string(7) "γ" ["Î´"]=&gt; string(7) "δ" ["Îµ"]=&gt; string(9) "ε" ["Î¶"]=&gt; string(6) "ζ" ["Î·"]=&gt; string(5) "η" ["Î¸"]=&gt; string(7) "θ" ["Î¹"]=&gt; string(6) "ι" ["Îº"]=&gt; string(7) "κ" ["Î»"]=&gt; string(8) "λ" ["Î¼"]=&gt; string(4) "μ" ["Î½"]=&gt; string(4) "ν" ["Î¾"]=&gt; string(4) "ξ" ["Î¿"]=&gt; string(9) "ο" ["Ï€"]=&gt; string(4) "π" ["Ï"]=&gt; string(5) "ρ" ["Ï‚"]=&gt; string(8) "ς" ["Ïƒ"]=&gt; string(7) "σ" ["Ï„"]=&gt; string(5) "τ" ["Ï…"]=&gt; string(9) "υ" ["Ï†"]=&gt; string(5) "φ" ["Ï‡"]=&gt; string(5) "χ" ["Ïˆ"]=&gt; string(5) "ψ" ["Ï‰"]=&gt; string(7) "ω" ["Ï‘"]=&gt; string(10) "ϑ" ["Ï’"]=&gt; string(7) "ϒ" ["Ï–"]=&gt; string(5) "ϖ" ["â€‚"]=&gt; string(6) " " ["â€ƒ"]=&gt; string(6) " " ["â€‰"]=&gt; string(8) " " ["â€Œ"]=&gt; string(6) "‌" ["â€"]=&gt; string(5) "‍" ["â€Ž"]=&gt; string(5) "‎" ["â€"]=&gt; string(5) "‏" ["â€“"]=&gt; string(7) "–" ["â€”"]=&gt; string(7) "—" ["â€˜"]=&gt; string(7) "‘" ["â€™"]=&gt; string(7) "’" ["â€š"]=&gt; string(7) "‚" ["â€œ"]=&gt; string(7) "“" ["â€"]=&gt; string(7) "”" ["â€ž"]=&gt; string(7) "„" ["â€ "]=&gt; string(8) "†" ["â€¡"]=&gt; string(8) "‡" ["â€¢"]=&gt; string(6) "•" ["â€¦"]=&gt; string(8) "…" ["â€°"]=&gt; string(8) "‰" ["â€²"]=&gt; string(7) "′" ["â€³"]=&gt; string(7) "″" ["â€¹"]=&gt; string(8) "‹" ["â€º"]=&gt; string(8) "›" ["â€¾"]=&gt; string(7) "‾" ["â„"]=&gt; string(7) "⁄" ["â‚¬"]=&gt; string(6) "€" ["â„‘"]=&gt; string(7) "ℑ" ["â„˜"]=&gt; string(8) "℘" ["â„œ"]=&gt; string(6) "ℜ" ["â„¢"]=&gt; string(7) "™" ["â„µ"]=&gt; string(9) "ℵ" ["â†"]=&gt; string(6) "←" ["â†‘"]=&gt; string(6) "↑" ["â†’"]=&gt; string(6) "→" ["â†“"]=&gt; string(6) "↓" ["â†”"]=&gt; string(6) "↔" ["â†µ"]=&gt; string(7) "↵" ["â‡"]=&gt; string(6) "⇐" ["â‡‘"]=&gt; string(6) "⇑" ["â‡’"]=&gt; string(6) "⇒" ["â‡“"]=&gt; string(6) "⇓" ["â‡”"]=&gt; string(6) "⇔" ["âˆ€"]=&gt; string(8) "∀" ["âˆ‚"]=&gt; string(6) "∂" ["âˆƒ"]=&gt; string(7) "∃" ["âˆ…"]=&gt; string(7) "∅" ["âˆ‡"]=&gt; string(7) "∇" ["âˆˆ"]=&gt; string(6) "∈" ["âˆ‰"]=&gt; string(7) "∉" ["âˆ‹"]=&gt; string(4) "∋" ["âˆ"]=&gt; string(6) "∏" ["âˆ‘"]=&gt; string(5) "∑" ["âˆ’"]=&gt; string(7) "−" ["âˆ—"]=&gt; string(8) "∗" ["âˆš"]=&gt; string(7) "√" ["âˆ"]=&gt; string(6) "∝" ["âˆž"]=&gt; string(7) "∞" ["âˆ "]=&gt; string(5) "∠" ["âˆ§"]=&gt; string(5) "∧" ["âˆ¨"]=&gt; string(4) "∨" ["âˆ©"]=&gt; string(5) "∩" ["âˆª"]=&gt; string(5) "∪" ["âˆ«"]=&gt; string(5) "∫" ["âˆ´"]=&gt; string(8) "∴" ["âˆ¼"]=&gt; string(5) "∼" ["â‰…"]=&gt; string(6) "≅" ["â‰ˆ"]=&gt; string(7) "≈" ["â‰ "]=&gt; string(4) "≠" ["â‰¡"]=&gt; string(7) "≡" ["â‰¤"]=&gt; string(4) "≤" ["â‰¥"]=&gt; string(4) "≥" ["âŠ‚"]=&gt; string(5) "⊂" ["âŠƒ"]=&gt; string(5) "⊃" ["âŠ„"]=&gt; string(6) "⊄" ["âŠ†"]=&gt; string(6) "⊆" ["âŠ‡"]=&gt; string(6) "⊇" ["âŠ•"]=&gt; string(7) "⊕" ["âŠ—"]=&gt; string(8) "⊗" ["âŠ¥"]=&gt; string(6) "⊥" ["â‹…"]=&gt; string(6) "⋅" ["âŒˆ"]=&gt; string(7) "⌈" ["âŒ‰"]=&gt; string(7) "⌉" ["âŒŠ"]=&gt; string(8) "⌊" ["âŒ‹"]=&gt; string(8) "⌋" ["âŒ©"]=&gt; string(6) "⟨" ["âŒª"]=&gt; string(6) "⟩" ["â—Š"]=&gt; string(5) "◊" ["â™ "]=&gt; string(8) "♠" ["â™£"]=&gt; string(7) "♣" ["â™¥"]=&gt; string(8) "♥" ["â™¦"]=&gt; string(7) "♦" }
  </code></pre>
  
  <p><strong>EDIT 3:</strong></p>
  
  <p>Just for full disclosure, here is a test file I have set up for the purposes of figuring this out. Currently, all entities display correctly, but accented characters display as <code>�</code>.</p>
  
  <pre><code>&lt;?php
  
  header('Content-Type: text/html; charset=UTF-8');
  
  require_once('_RESOURCES/simple_html_dom.php');
  
  $url = 'http://fanfiction.portkey.org/index.php?act=read&amp;storyid=1585&amp;chapterid=&amp;agree=1';
  
  function tidyHTML($html) {
      ob_start();
      $tidy = new tidy;
      $config = array('indent' =&gt; true, 'output-xhtml' =&gt; false, 'wrap' =&gt; 200, 'clean' =&gt; false, 'show-body-only' =&gt; true);
      $tidy-&gt;parseString($html, $config, 'utf8');
      $tidy-&gt;cleanRepair();
      $input = $tidy;
      return $input;
  }
  
  function filter($html) {
      $html = preg_replace('~&gt;\s+&lt;~', '&gt;&lt;', $html);
      $html = preg_replace('/&lt;\/b&gt;\s?&lt;b&gt;/', '', $html);
      $html = preg_replace('/&lt;\/i&gt;\s?&lt;i&gt;/', '', $html);
      $html = str_replace('&lt;br&gt;', '', $html);
      $output = $html;
      return $output;
  }
  
  $page_html = file_get_html($url);
  $chapter_html = $page_html-&gt;find('td[class="story"]', 0);
  foreach ($chapter_html-&gt;find('center') as $node) { $node-&gt;outertext = ''; }
  
  $entities = html_entity_decode($chapter_html-&gt;innertext, ENT_QUOTES, 'UTF-8');
  
  echo tidyHTML(filter($entities));
  
  // var_dump(get_html_translation_table(HTML_ENTITIES));
  
  ?&gt;
  </code></pre>