title: How to build a lookup map in Spark Streaming?
body: <p>What is the best way to maintain application state in a spark streaming application?</p>
  
  <p>I know of two ways :</p>
  
  <ol>
  <li>use "Union" operation to append to the lookup RDD and persist it after each union.</li>
  <li>save the state in a file or database and load it in the start of each batch.</li>
  </ol>
  
  <p>My question is from the performance perspective which one is better ? Also, is there a better way to do this?</p>