title: Cannot change global variables within Spark foreach function body
body: <p>I need to filter through a very large text file(500GB) and export the results with Spark.</p>
  
  <p>The problem here is that it will be too big to fit everything in memory, so I have to use a global buffered writer, which gets flushed about every 500000 records.</p>
  
  <p>Intuitively, I used RDD's foreach function and passed a function to it, like (in pyspark):</p>
  
  <pre><code># global definitions
  def fn(x);
      global counter, writer
      # some logic to generate data
      if counter == 500000:
          writer.write(data)
          writer.flush_buf()
          counter = 1
      else:
          counter = counter + 1
  txt_file.foreach(fn)
  # write final data
  </code></pre>
  
  <p>However, here somehow the variable counter is not being updated.</p>
  
  <p>I initially thought it might have something to do with the python-scala bridge, and rewrote some test code with scala:</p>
  
  <pre><code>var value = 10
  def fn(x: Int) = {
      value = 20
  }
  sc.parallelzie(Seq(1, 2, 3, 4, 5), 5).foreach(fn)
  println(value)
  </code></pre>
  
  <p>To my surprise, I still get value=10. I wonder why I'm getting this strange behavior here and what's the correct way of doing it?</p>
  
  <p>Thanks!</p>