title: Using a broadcast variable in a loop in Spark with updated values
body: <p>I am using a broadcast variable in a loop like the following (to make it short, I show it with a kind of psudocode, not exact syntax in Java):</p>
  
  <pre><code>Broadcast&lt;List&lt;E&gt;&gt; brdList = jsc.broadcast(myVariable);
  JavaRDD&lt;myType&gt; rdd = rawRdd.map(f(brdList.value()));  
  List&lt;E&gt; updatedBrdList = rdd.map(g).collect();
  brdList.unpersist();
  
  int itr = 1000;
  while (itr != 0){
     Broadcast&lt;List&lt;E&gt;&gt; brdNewList(updatedBrdList);
     rdd = rdd.map(f(brdNewList.value()));
     updatedBrdList = rdd.map(g).collect();
     itr--;
  }
  </code></pre>
  
  <p>Is this usage a valid form of using broadcast variable? Does the brdNewList occupy one location in memory or in each iteration new space is occupied and a new copy is created?</p>
  
  <p>With few iterations (~ &lt;100), it works fine, but with larger iterations, it gives the following error:
  <a href="https://i.stack.imgur.com/BMWZ9.png" rel="nofollow noreferrer">the error with using broadcast variable in Spark</a></p>
  
  <p>Is there any way to play around it and make it works? The value of the broadcast variable is necessary to be accessed through all nodes in each iteration.</p>
  
  <p>Is this related to driver memory? or there are some computations in executors (workers)? (I am running my code on a cluster with 5 nodes.) </p>
  
  <p>Any help is appreciated!</p>