title: Avoid lazy evaluation of code in spark without cache
body: <p>How can I avoid lazy evaluation in spark. I have a data frame which needs to be populated at once, since I need to filter the data on the basis of random number generated for each row of data frame, say if random number generated > 0.5, it will be filtered as dataA and if random number generated &lt; 0.5 it will be filtered as dataB.</p>
  
  <pre><code>val randomNumberDF = df.withColumn("num", Math.random())
  val dataA = randomNumberDF.filter(col("num") &gt;= 0.5)
  val dataB = randomNumberDF.filter(col("num") &lt; 0.5)
  </code></pre>
  
  <p>Since spark is doing lazy eval, while filtering there is no reliable distribution of rows which are being filtered as dataA and dataB(sometimes same row is being present in both dataA and dataB)</p>
  
  <p>How can I avoid this re-computation of "num" column, I have tried using "cache", which worked, but given my data size is going to be big, I am ruling out that solution. 
  I have also tried using other actions on the randomNumberDF, like :</p>
  
  <pre><code>count
  rdd.count
  show
  first
  </code></pre>
  
  <p>these didn't solve the problem.</p>
  
  <p>Please suggest something different from cache/persist/writing data to HDFS and again reading it as solution.</p>
  
  <p>References I have already checked : </p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/46810203/how-to-force-spark-to-avoid-dataset-re-computation">How to force spark to avoid Dataset re-computation?</a></li>
  <li><a href="https://stackoverflow.com/questions/56214599/how-to-force-spark-to-only-execute-a-transformation-once">How to force Spark to only execute a transformation once?</a></li>
  <li><a href="https://stackoverflow.com/questions/39380923/how-to-force-spark-to-evaluate-dataframe-operations-inline">How to force Spark to evaluate DataFrame operations inline</a></li>
  </ul>