title: Creating a large dictionary in pyspark
body: <p>I am trying to solve the following problem using pyspark. 
  I have a file on hdfs in the format which is a dump of lookup table.</p>
  
  <pre><code>key1, value1
  key2, value2
  ...
  </code></pre>
  
  <p>I want to load this into python dictionary in pyspark and use it for some other purpose. So I tried to do:</p>
  
  <pre><code>table = {}
  def populateDict(line):
      (k,v) = line.split(",", 1)
      table[k] = v
  
  kvfile = sc.textFile("pathtofile")
  kvfile.foreach(populateDict)
  </code></pre>
  
  <p>I found that table variable is not modified. So, is there a way to create a large inmemory hashtable in spark?</p>