title: How to check if data is cached in dataframe or not yet cached due to lazy execution in Pyspark?
body: <p>My question is little different from other question I could find on stack overflow. I need to know if the data is already retrieved and stored in a dataframe or if that is yet to happen</p>
  <p>I am doing something like this</p>
  <pre><code>df1=spark.table(&quot;sourceDB.Table1&quot;)
  df1.cache()
  </code></pre>
  <p>Now, as you might be aware, data is not read yet from the source table due to lazy execution. So I need to have an expression here that says the result as &quot;False&quot; at this point.</p>
  <p>After sometime, I am doing some operation that requires data to be retrieved from source. For example.</p>
  <pre><code>df1.groupBy(&quot;col3&quot;).agg(sum(&quot;col1&quot;).alias(&quot;sum_of_col1&quot;)).select(&quot;sum_of_col1&quot;,&quot;col3&quot;).filter(&quot;sum_of_col1 &gt;= 100&quot;).show()
  </code></pre>
  <p>At this point, data must have been read and stored in cache for df1. So I need to have an expression here that says the result as &quot;True&quot; at this point.</p>
  <p>Is there anyway we can achieve this? I believe df1.is_cached will not help in this situation</p>