title: Strip non-printable characters using Hadoop Map-Reduce
body: <p>I am trying to process a HDFS file which has non-printable chracters. I wish to strip these characters out using MapReduce. </p>
  
  <p>I have tried using Pig TextLoader and MR TextInputFormat (IN MR program), and it results to split of a record into multiple from the position where it encounters a non-printable characters. Below is the sample data:</p>
  
  <p>===Data==(2 records)=</p>
  
  <pre><code> 4614:2011-12-20-08.45.08.169176^2011-12-20-18.15.08.100008^597^0^57^ZUKA^Grase^^^Grase,Dr^^^N^N^N^Dr^KG^ONLY INFORMATION ENTERED^UNKNOWN^0     ^      ^^^611190362
  ï¿½^0^^^^^^0^Grase,Dr^^^, ,^^^^^^597^^^&lt;fnm&gt;DR&lt;/fnm&gt;&lt;lnm&gt;GRASE&lt;/lnm&gt;^^^^^^^^SINGLE^0^0
  6063:2010-07-04-04.00.00.100001^2010-07-04-04.01.00.180144^017^0^095^WEETE    ^Wien^^^Wien,Colock^^^N^N^N^Colock^KG^ONLY INFORMATION ENTERED^UNKNOWN^0     ^      ^295111915^^ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½9905^0^^^^^^0^Wien,Colock^40001 KIN RD^300 CAMORE ST^500 BLACK AVE^Woesfield, HA, 43723.^John Ball^^^25719110^617^ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½9905^^&lt;fnm&gt;COLOCK&lt;/fnm&gt;&lt;lnm&gt;WIEN&lt;/lnm&gt;^^^^^^^^SINGLE^0^0
  </code></pre>
  
  <p>[In less editor, the first record for column with special characters looks as: </p>
  
  <p><code>611190362^M&lt;EF&gt;&lt;BF&gt;&lt;BD&gt;</code>]</p>
  
  <p>In vi or less the first record comes in one line, but while reading in MR or pig this record gets splitted due to presence of these characters.</p>
  
  <p>I want to avoid the record splitting into new line while reading data from HDFS and further wish to process these records to get rid of these special characters.</p>
  
  <p>Here is what i have tried with basic UDF (snippet below). Though, the program is striping the charcters > 0x80 but performs stripping on splitted record.</p>
  
  <p>Any help / pointers would be appreciated!! </p>
  
  <pre><code>        /*
       * 
       * Pig Code:
       * register '/basepath/udf/DIF.jar'
      rel1 = LOAD '/user/home/test' USING USING TextLoader();
      rel2 = FOREACH rel1 GENERATE StripNonPrintable(s) as recordline;
      dump rel2;
  
       * 
       */
  
    //Imports    
  
      public class StripNonPrintable extends EvalFunc&lt;String&gt;
      {
           public String exec(Tuple input) throws IOException {
               if (input == null || input.size() == 0)
                   return null;
               try{
  
                      String s = new String();
                       s = (String)input.get(0);
                      //s = "2001-12-20-08.45.08.169176^2001-12-20-08.45.08.131408^597^0^57^ZUCKA^Grase^^^Grase,Dr^^^N^N^N^Dr^KG^ONLY INFORMATION ENTERED^UNKNOWN^0     ^      ^^^6785790362ï¿½^0^^^^^^0^Grase,Dr^^^, ,^^^^^^597^^^&lt;fnm&gt;DR&lt;/fnm&gt;&lt;lnm&gt;GRASE&lt;/lnm&gt;^^^^^^^^SINGLE^0^0";
  
                      int length = s.length();
                      char[] oldChars = new char[length];
                      s.getChars(0, length, oldChars, 0);
                      int newLen = 0;
                      for (int j = 0; j &lt; length; j++) {
                          char ch = oldChars[j];
                          if (ch &lt; 0x80  ) {
                              oldChars[newLen] = ch;
                              newLen++;
                          }
                      }
                      s = new String(oldChars, 0, newLen);
  
                      //System.out.println("New String = \n " + s);
  
                      return s;
               }catch(Exception e){
                   return null ;
               }
           }
  
       }
  </code></pre>