title: "Illegal Byte sequence" error while using shell commands in mac bash terminal
body: <p>Getting "illegal byte sequence" error while trying to extract non English characters from a large file in MacOS bash shell.
  This is the script that I am trying to use:</p>
  
  <pre><code>sed 's/[][a-z,0-9,A-Z,!@#\$%^&amp;*(){}":/_-|. -][\;''=?]*//g' &lt; $1 &gt;Abhineet_extract1.txt;
  sed 's/\(.\)/\1\
  /g' &lt;Abhineet_extract1.txt | sort | uniq |tr -d '\n' &gt;&amp;1;
  rm Abhineet_extract1.txt;
  </code></pre>
  
  <p>and here is the error that I am getting: </p>
  
  <pre><code>uniq: stdin: Illegal byte sequence
  </code></pre>
  
  <p>'+?</p>