title: Scope of Spark's `persist` or `cache`
body: <p>I am confused about RDD's scoping in Spark.</p>
  
  <p>According to <a href="https://stackoverflow.com/questions/32911128/scope-of-cached-rdds">this thread</a></p>
  
  <blockquote>
    <p>Whether an RDD is cached or not is part of the mutable state of the RDD object. If you call rdd.cache it will be marked for caching from then on. It does not matter what scope you access it from.</p>
  </blockquote>
  
  <p>So, if I defined a function with a new rdd created inside, for example (python code)</p>
  
  <pre><code># there is an rdd called "otherRdd" outside the function
  
  def myFun(args):
      ...
      newRdd = otherRdd.map(some_function)
      newRdd.persist()
      ...
  </code></pre>
  
  <p>Will the <code>newRdd</code> lives in the global namespace? or it is only visible inside the environment of <code>myFun</code>? </p>
  
  <p>If it is only visible inside the environment of <code>myFun</code>, after <code>myFun</code> finishes execution, will Spark automatically <code>unpersist</code> the <code>newRdd</code>? </p>