title: Apache Spark: Unexpected filter results
body: <p>I am working with Apache Spark v 1.2 in local mode. I have created an RDD and persisted it in memory. The Spark Web UI shows that 85% of this RDD is stored in memory. I have a feature/variable in the RDD that has values of 0,1 as demonstrated by the result I get by running the script below:</p>
  
  <pre><code>In[96]: flagged.map(lambda x:(x[14],1)).reduceByKey(lambda x,y:x+y).collect()
  
  Out[96]: [(0, 637981), (1, 272958)]
  </code></pre>
  
  <p>Also when I do a flagged.count() the number is the sum of the two values  i.e. 637981+272958 = 910939</p>
  
  <p>Now when I run a filter based on this, I do not get the same counts:</p>
  
  <pre><code>In[97]:  flagged.filter(lambda x: x[14]==0).count()
  
  Out[97]:  637344
  
  In[97]:  flagged.filter(lambda x: x[14]==1).count()
  
  Out[97]:  272988
  </code></pre>
  
  <p>I am struggling to understand that why do the numbers produced from the filtered RDDs do not match that of the reduceByKey method.</p>