title: Are DataFrames created every time using DAG, when the DF is referenced multiple time in spark code(if we don't use persist)?
body: <p>I faced this question in an interview. The sequence of operations are as below.</p>
  <p><strong>Read the csv file and create a DF.</strong>
  DF = spark.read().format(&quot;csv&quot;).csv(&quot;/path/file.csv&quot;)</p>
  <p><strong>Filter the data based on some condition.</strong>
  DF.filter(....)</p>
  <p><strong>perform a count operation.</strong>
  DF.count()</p>
  <p><strong>Save the Data in file/Table.</strong>
  DF.saveAsTable()</p>
  <p>The question was how many transformations and actions are there and how many times the csv file is read to create a DF?</p>
  <p>I am curious to understand the below</p>
  <p>Once the file is read, won't the DF stay in memory till the spark job is complete?
  If the DF is not being used in further steps, we don't mind even if it is removed from memory.
  But if it is being used in further steps, won't it stay in the memory till the step where it is last used?</p>
  <p>Does the file read and DF creation happens every time whenever each action is triggered? Why would this be required at all?</p>
  <p>I understand that the file is read only when a action is performed.</p>
  <p>Now when the count() action is called first, the file has been read into memory and a DF is built. so I expect it to stay in the memory. But why will saveAsTable() action read the file again and reconstruct the DF again using DAG?</p>
  <p>If it is re-evaluating the DF(using DAG) everytime we use the DF reference in our code, is it not time consuming to reconstruct the DF every time?
  If we have to avoid this reconstruction process everytime and if we start persisting the DF's, will we not end up occupying all the memory and spilling some data to disk and slow down the spark application?</p>
  <p>Can anyone explain whats going on?</p>