title: Turn RDD into Broadcast Dictionary for lookup
body: <p>What I have so far is: </p>
  
  <pre><code>lookup = sc.textFile("/user/myuser/lookup.asv")
  lookup.map(lambda r: r.split(chr(1)) )
  </code></pre>
  
  <p>And now I have a RDD looks like</p>
  
  <pre><code>[
      [filename1, category1],
      [filename2, category2],
      ...
      [filenamen, categoryn]
  ]
  </code></pre>
  
  <p>How can I turn that RDD into a broadcasted dictionary like:</p>
  
  <pre><code>{filename1: category1, filename2: category2, ...}
  </code></pre>
  
  <p>This is what I have tried but not working: </p>
  
  <pre><code>&gt;&gt;&gt; broadcastVar = sc.broadcast({})
  &gt;&gt;&gt; data = sc.parallelize([[1,1], [2,2], [3,3], [4,4]])
  &gt;&gt;&gt; def myfunc(x):
  ...     broadcastVar[str(x[0])] = x[1]
  ... 
  &gt;&gt;&gt; result = data.map(myfunc)
  &gt;&gt;&gt; broadcastVar
  &lt;pyspark.broadcast.Broadcast object at 0x7f776555e710&gt;
  &gt;&gt;&gt; broadcastVar.value
  {}
  &gt;&gt;&gt; result.collect()
  ...
  ERROR: TypeError: 'Broadcast' object does not support item assignment
  ...
  &gt;&gt;&gt; broadcastVar.value
  {}
  </code></pre>
  
  <hr>
  
  <p>For more information about why I am building this huge lookup variable, read this: </p>
  
  <p>This is a followup question of this <a href="https://stackoverflow.com/questions/33138129/spark-using-pyspark-read-images">one</a>. </p>
  
  <p>I have two tables where </p>
  
  <p>table1: a very wide (25K columns and 150K rows) table where each column contains the pixel info and the first column is the filename of the input image file. </p>
  
  <p>table2: a TSV (tab delimited file) file that has 3 million rows and each row contains the image file name and the product category of the image. </p>
  
  <p>Speaking in SQL, I need to do a inner join on those two tables on the filename so I can label the image data for later on machine learning. </p>
  
  <p>It is not realistic to do it in any sort of SQL because you have to create a table for table1 which has 25K columns, the create table syntax will be ridiculous long. </p>
  
  <p>Then I am thinking about creating a lookup variable using table2 and maybe make it a broadcast variable where the key is the filename and the value is the product category. </p>