title: How can I filter an RDD based on length of lines in the dataset.?
body: <blockquote>
    <p>I want to filter out an RDD which is created from a dataset, based on
    length of line Using: Pyspark shell </p>
    
    <p>My data file looks like this</p>
  </blockquote>
  
  <pre><code>&gt; fzDTn342L3Q   djjohnnykey 599 Music   185 1005    3.67    3   1   KDrJSNIGNDQ MacjQFNVLlQ oZ6f2vaH858 fYSjMDNa4S8 JUatPzf_eSc QfBFl7kU35c rG-rQ-YGdSA kOq6sFmoUr0 IRj1IABVBis AVsZ0VH3eN4 r1pS_4qouUc YgaNW1KRgK4 ZlGdVR7mBy4 nKFLE3DX4OQ EtQjN6CQeCc afe-0VY4YiI ekV5NseEdy8 IQs6CrER5fY jTLcoIxMI-E yfvW1ITcMpM
  &gt; 
  &gt; kOq6sFmoUr0   djjohnnykey 599 Music   113 992 0   0   1   MacjQFNVLlQ fYSjMDNa4S8 4vso1y_-cvk 8BwAX6YBx3E QeUQyf8H7vM jmc21-Nhewg hZUU2-UBaGk SaLaotssH0w PUlcrBaYpwI tjIK2xop4L0 BNlL15OYnFY _pzP7OLInjk 4daGJ6TMcp4 _8jM9R-1yRk KDrJSNIGNDQ oZ6f2vaH858 JUatPzf_eSc QfBFl7kU35c rG-rQ-YGdSA fzDTn342L3Q
  </code></pre>
  
  <blockquote>
    <p>Here the 4th column is category. Some of the lines in the data file
    does not contain this field and hence less in length.  This motivates
    me to filter out the data set based on this criteria and further form
    RDDs on that set of data which has category.</p>
    
    <p>I have tried to create initial RDD from the dataset.</p>
  </blockquote>
  
  <pre><code>&gt;&gt;&gt; data="/Users/sk/Documents/BigData/0222/0.txt"
  &gt;&gt;&gt; input = sc.textFile(data)
  </code></pre>
  
  <p>Now I am splitting by tab and saving in lines RDDS</p>
  
  <pre><code>&gt;&gt;&gt; lines = input.map(lambda x: (str(x.split('\t'))))
  </code></pre>
  
  <p>After this I want to filter out the lines whose length is less than 3.</p>
  
  <pre><code>&gt;&gt;&gt; data="/Users/sk/Documents/BigData/0222/1.txt"
  &gt;&gt;&gt; input = sc.textFile(data)
  &gt;&gt;&gt; lines = input.map(lambda x: (str(x.split('\t'))))
  &gt;&gt;&gt; lines.count()
  3169
  
  &gt;&gt;&gt; newinput=input.filter(lambda x: len(x)&gt;3)
  &gt;&gt;&gt; newinput.count()
  3169
  </code></pre>
  
  <blockquote>
    <p>Following this it does not change anything in my rdd. Can anyone
    please help out.</p>
  </blockquote>