title: Is there any way to link the rows of an RDD like a linked list in PySpark?
body: <p>I have a large RDD like the following:</p>
  <pre><code>[(3, (2, 'Child')), (2, (1, 'Parent')), (1, (None, 'Root'))]
  </code></pre>
  <p>where the key of each element is a unique index for that element and 1st element in the value pair signifies the index of its parent element.</p>
  <p>Now, I need to transform this RDD into the following:</p>
  <pre><code>[(3, (2, 'Child Parent Root')), (2, (1, 'Parent Root')), (1, (None, 'Root'))]
  </code></pre>
  <p>where the 2nd element in the value pair for each item will be modified such that it has the concatenation of all the values up to its root ancestor.</p>
  <p>The no. of levels is not fixed and can be up to 256. So, RDD joining is probably not an option.</p>
  <p>I tried saving the parent values into an accumulator dictionary and broadcasting the dict but the dict becomes too large (12GB) and gets OOM. Any idea on how can I achieve this efficiently?</p>